SmoothDeblock (under development) [Archive]

View Full Version : SmoothDeblock (under development)

redfordxx

23rd May 2006, 21:08

Hi
hope you are not bored with my deblocking attempts... Here is my latest trial:

Remember SmoothD? I tried to figure out how it works and made similar script (that§s why I borrowed the name) but adaptive... and slow

I made shifted DCTFilter applications to remove high frequencies. This caused removing the blocking on original block borders (shifted into middles of the blocks) but introduced new blocking on the new borders. So, when I shift the clips back, I have (for example) four clips with blocking on diferent places. I merge the clips so that I take only the inner part of the block (without blocking edge).
For this I use weighting mask made by function WeightMatrixClip which is clip made of all the same 8x8 blocks with values 255 in the middle and going down to borders. Additional functionWeightMatrixSum delivers the average value of the mask clip which is later used to normalize the weighted average.
This all delivers function SmoothDeblockLayer and it is possible to choose the density, quant, weightmask.
density = number of shifts on each axis per 8pixels - possible values are 2 or 4. (so 4 means 16 shifts per 8x8 block), 4 produces sharper image and less artifacts on hi quants but is more than 4times slower than 2, which means only 4 shifts...)
quant = 0,...,7 number of coeficients removed by DCTfilter. 0 makes no change to the picture.
weightmask = WeightMatrixClip mask together with WeightMatrixSum controls how the shifted clips are merged.
So, SmoothDeblockLayer returns deblocked clip on certain quant.

I can have 7 different deblocked clips according to quant. I want to merge some of them depending on detail presence.

Then, my assumption (not 100% correct) was, blocking is where is hi quantization, so, let's detect the quantization. I detect the quantization with DCTFilter again on certain quant again. That's function QuantDiff. It produces the mask of "what could be removed with lowpass on certain quant".

From this mask, the function AdaptionMask produces mask for merging the deblocked "layers". It is just some values increasing, blockaveraging and blurring depending on given parameters. This is part wherer I am not sure. Now it is block oriented but maybe the mask should more precisely follow the detail of the clip.

Finally the SmoothDeblock function puts everything together. Now it is fixed for 3 deblocking layer, later is should be parametrizable.
It is pretty slow now and it will be probably always. I think still there is significant speedup possible when reorganizing the script, but I wanna keep the script structured like this until it is finished, to keep it a little readable.find it in posts 2, 16 and 17
Important: source has to be perfecly deinterlaced, otherwise the deblocker does nothing

redfordxx

23rd May 2006, 21:14

You need
Arrays and Loops.avsi (http://forum.doom9.org/showthread.php?p=780778#post780778)
Patterns.avsi
function Matrix (clip clp, int "uv")
{
uv = default(uv,3)

unit=blankclip(clp,width=4,height=4,pixel_type="YV12")
row=stackhorizontal( unit.mt_lut(y=0,u=2,v=2),
\ unit.mt_lut(y=-1,u=2,v=2),
\ unit.mt_lut(y=-2,u=2,v=2),
\ unit.mt_lut(y=-3,u=2,v=2),
\ unit.mt_lut(y=-4,u=2,v=2),
\ unit.mt_lut(y=-5,u=2,v=2),
\ unit.mt_lut(y=-6,u=2,v=2),
\ unit.mt_lut(y=-7,u=2,v=2))
square=stackvertical( row.mt_lut("x 0 +",y=3,u=2,v=2),
\ row.mt_lut("x 8 +",y=3,u=2,v=2),
\ row.mt_lut("x 16 +",y=3,u=2,v=2),
\ row.mt_lut("x 24 +",y=3,u=2,v=2),
\ row.mt_lut("x 32 +",y=3,u=2,v=2),
\ row.mt_lut("x 40 +",y=3,u=2,v=2),
\ row.mt_lut("x 48 +",y=3,u=2,v=2),
\ row.mt_lut("x 56 +",y=3,u=2,v=2))
block=square#.mt_lut("x 4 *",y=3,u=2,v=2)
block4=stackvertical(stackhorizontal(block,block),stackhorizontal(block,block)).pointresize(16,16)
block4 = (uv!=3) ? block4.mt_lut(y=3,u=uv,v=uv)
\ : YtoUV(block4.crop(0,0,8,8),block4.crop(0,0,8,8),block4)
return(block4.trim(1,1))
}

function PtrnFormat(clip clp, clip form)
{
fh=form.height
fw=form.width
clp=StackHorizontal(clp,clp,clp,clp,clp,clp,clp,clp,clp,clp,clp)
clp=StackVertical(clp,clp,clp,clp,clp,clp,clp,clp,clp)
clp=(fw > 160) ? StackHorizontal(clp,clp,clp,clp) : clp
clp=(fh > 128) ? StackVertical(clp,clp,clp,clp) : clp
clp=(fw > 640) ? StackHorizontal(clp,clp,clp) : clp
clp=(fh > 512) ? StackVertical(clp,clp,clp) : clp
clp=clp.crop(0,0,fw,fh)

return(clp.loop(framecount(form)))
}

function PtrnVertical(clip clp, int "uv", int "v0", int "v1", int "v2", int "v3", int "v4", int "v5", int "v6", int "v7")
{
v1 = default(v1,255-v0)
v2 = default(v2,v1)
v3 = default(v3,v2)
v4 = default(v4,v3)
v5 = default(v5,v2)
v6 = default(v6,v1)
v7 = default(v7,v0)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 8 % 0 == "+String(v0)+" "+String(v1)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 2 == "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 3 == "+String(v3)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 4 == "+String(v4)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 5 == "+String(v5)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 6 == "+String(v6)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 8 % 7 == "+String(v7)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}

function PtrnHorizontal(clip clp, int "uv", int "v0", int "v1", int "v2", int "v3", int "v4", int "v5", int "v6", int "v7")
{
v1 = default(v1,255-v0)
v2 = default(v2,v1)
v3 = default(v3,v2)
v4 = default(v4,v3)
v5 = default(v5,v2)
v6 = default(v6,v1)
v7 = default(v7,v0)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 4 - 8 / 8 % 0 == "+String(v0)+" "+String(v1)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 2 == "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 3 == "+String(v3)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 4 == "+String(v4)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 5 == "+String(v5)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 6 == "+String(v6)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 7 == "+String(v7)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}

function PtrnBlock(clip clp, int "uv", int "v0", int "v1", int "v2", int "v3")
{
v1 = default(v1,255-v0)
v2 = default(v2,v1)
v3 = default(v3,v2)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 4 - 8 / 8 % 1 >= x 4 - 8 / 8 % 6 <= & x 8 % 1 >= x 8 % 6 <= & & "+String(v1)+" "+String(v0)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 2 >= x 4 - 8 / 8 % 5 <= & x 8 % 2 >= x 8 % 5 <= & & "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 3 >= x 4 - 8 / 8 % 4 <= & x 8 % 3 >= x 8 % 4 <= & & "+String(v3)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}

function PtrnChess(clip clp, int "uv", int "v0", int "v1", int "v2", int "v3")
{
v1 = default(v1,255-v0)
v2 = default(v2,v0)
v3 = default(v3,v1)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 4 - 8 / 2 % 0 == x 2 % 0 == & "+String(v0)+" "+String(v1)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % 1 == x 2 % 1 == & "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % 1 == x 2 % 0 == & "+String(v3)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}

function PtrnSawH(clip clp, int "uv", string "shift", int "v0", int "v1", int "v2", int "v3", int "v4", int "v5", int "v6", int "v7")
{
v1 = default(v1,255-v0)
v2 = default(v2,v1)
v3 = default(v3,v2)
v4 = default(v4,v3)
v5 = default(v5,255-v4)
v6 = default(v6,v5)
v7 = default(v7,v6)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 4 - 8 / 8 % 0 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 0 == x 1 + 2 % "+shift+" 2 % == & | "+String(v0)+" "+String(v1)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 2 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 2 == x 1 + 2 % "+shift+" 2 % == & | "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 3 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 3 == x 1 + 2 % "+shift+" 2 % == & | "+String(v3)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 4 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 4 == x 1 + 2 % "+shift+" 2 % == & | "+String(v4)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 5 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 5 == x 1 + 2 % "+shift+" 2 % == & | "+String(v5)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 6 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 6 == x 1 + 2 % "+shift+" 2 % == & | "+String(v6)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 8 % 7 == x 2 % "+shift+" 2 % == & x 4 + 8 / 8 % 7 == x 1 + 2 % "+shift+" 2 % == & | "+String(v7)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}

function PtrnSawV(clip clp, int "uv", string "shift", int "v0", int "v1", int "v2", int "v3", int "v4", int "v5", int "v6", int "v7")
{
v1 = default(v1,255-v0)
v2 = default(v2,v1)
v3 = default(v3,v2)
v4 = default(v4,v3)
v5 = default(v5,255-v4)
v6 = default(v6,v5)
v7 = default(v7,v6)

mtrx=Matrix(clp,uv)
ptrn=mt_lutxy(mtrx,mtrx,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 0 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 0 == & | "+String(v0)+" "+String(v1)+" ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 2 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 2 == & | "+String(v2)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 3 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 3 == & | "+String(v3)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 4 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 4 == & | "+String(v4)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 5 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 5 == & | "+String(v5)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 6 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 6 == & | "+String(v6)+" y ?",y=3,u=uv,v=uv)
ptrn=mt_lutxy(mtrx,ptrn,"x 4 - 8 / 2 % "+shift+" 2 % == x 8 % 7 == & x 4 + 8 / 2 % "+shift+" 2 % == x 1 + 8 % 7 == & | "+String(v7)+" y ?",y=3,u=uv,v=uv)

return(ptrn.PtrnFormat(clp))
}
and
function AverageBlock8(clip orig)
{
o1=orig.mt_convolution("0 0 0 0 0 0 0 1 1 1 1 1 1 1 1","0 0 0 0 0 0 0 1 1
1 1 1 1 1 1",y=3,u=3,v=3)
o1=o1.pointresize(o1.width/8,o1.height/8)
o1=o1.pointresize(o1.width*8,o1.height*8)
return(o1)
}

function VerticalLinearBlock(clip orig)
{
p1=PtrnHorizontal(orig,3,255,219,182,146,109,73,36,0)
top1=orig.PointResize(orig.width,orig.height/8)
top2=top1.PointResize(orig.width,orig.height)
shift1=orig.mt_convolution("1","0 0 0 0 0 0 0 0 0 0 0 0 0 0 1",y=3,u=3,v=3)
bottom1=shift1.PointResize(orig.width,orig.height/8)
bottom2=bottom1.PointResize(orig.width,orig.height)
mt_merge(bottom2,top2,p1,y=3,u=3,v=3)
}

function HorizontalLinearBlock(clip orig)
{
p1=PtrnVertical(orig,3,255,219,182,146,109,73,36,0)
top1=orig.PointResize(orig.width/8,orig.height)
top2=top1.PointResize(orig.width,orig.height)
shift1=orig.mt_convolution("0 0 0 0 0 0 0 0 0 0 0 0 0 0 1","1",y=3,u=3,v=3)
bottom1=shift1.PointResize(orig.width/8,orig.height)
bottom2=bottom1.PointResize(orig.width,orig.height)
mt_merge(bottom2,top2,p1,y=3,u=3,v=3)
}
+ DCTFilter
+ Average
which are
http://mywebpages.comcast.net/trbarry/DctFilter.zip
http://people.pwf.cam.ac.uk/mg262/posts/blend/Average_21Oct05.dll
from:
http://mywebpages.comcast.net/trbarry/downloads.htm
http://forum.doom9.org/showthread.php?t=100626&highlight=average

redfordxx

23rd May 2006, 21:28

#deblocks on defaults, ...
source.SmoothDeblock()

#deblocks on density 4, ...
source.SmoothDeblock(density=4,wieghtMX="16center255border0smooth_1",gain=10,pregain=10, pregainoffset=0)

#deblocks on quant 5 density 2, ...
source.SmoothDeblockLayer(5,2,orig.AddBorders(16,16,0,0).WeightMatrixClip("04center255border0smooth_1"),WeightMatrixSum("04center255border0smooth_1"))

#shows which detail would be destroyed by DCTFilter on quant 4
qd4=source.QuantDiff(4)

# makes block oriented mask based on clip from previous function
qd4.AdaptionMask(10,0,10,"1","1 1 1 1 1")

# the higher following values the mor detail and blocking is kept
pregainoffset= default(pregainoffset,-1)
pregain= default(pregain,10)
gain= default(gain,5)

krieger2005

25th May 2006, 02:31

+ DCTFilter
+ Average
Line 101: o1=o1.pointresize(o1.width/8,o1.height/8) -> o1=o1.pointresize(m8(o1.width/8),m8(o1.height/8))

I will look on this filter. Thanks

redfordxx

26th May 2006, 13:04

...-> o1=o1.pointresize(m8(o1.width/8),m8(o1.height/8))What do you mean by that?

krieger2005

26th May 2006, 14:12

I Got an error, because the Resize was not a 'mod 4' resize. So i just want to say, that you must change this to be sure to resize to a 'mod 8'-resolution (YV12-Limits).

foxyshadis

26th May 2006, 14:34

That will mess it up though, it makes some blocks now 1x2 instead of 1x1. (PointResize is picky that way.) You should pad borders to make it mod32 (so the output is mod4, doesn't need to be mod8). Then crop after resizing back up.

redfordxx

26th May 2006, 18:48

This function produces the weighting mask for the merging. Parameter clp is there to have same fps,size...

The masks can be named... my convention is that matrix with name 04* needs 4 shifted layers to average and obtain correct image, 16* needs 16 shifted layers to average and obtain correct image. One of advantages of the 16* matrix (for density=4) is that you can pick any numbers there and as long as it is symetric, it will work.

Example --- creation of matrix 04center255border0smooth_1:
For creating the matrix I use WeightMatrixClip function with heavy support of Patterns.avsi
function WeightMatrixClip{
#getting 8x8 matrix with pixel values 0...63 formated to size of original clip
mx1=clp.Matrix(3).PtrnFormat(clp)
#creating base matrix with color 0 at the border and 255 in the middle etc. (3 is for chroma processing as in mt)
mx=clp.PtrnBlock(3,0,50,200,255)

#each block "is made of 4 squares inside each other". Let's number them 0...3. (3 is in the middle)
#middle1 (correction of the side middle pixels on square no. 1 and setting on color 55. on mx1 the pixels are numbered 11,12,25,30...)
mx = mt_lutxy(mx,mx1,"y 12 == y 11 == y 25 == y 30 == y 33 == y 38 == y 51 == y 52 == | | | | | | | 55" x ?",y=3,u=3,v=3)
#diag1 (setting the on diagonal pixels on square no. 1 on color 25. on mx1 the pixels are numbered 9,14,49,54)
mx = mt_lutxy(mx,mx1,"y 9 == y 14 == y 54 == y 49 == | | | 25 x ?",y=3,u=3,v=3)
#diag2 (setting the on diagonal pixels on square no. 2 on color 130. on mx1 the pixels are numbered 18,21,42,45)
mx = mt_lutxy(mx,mx1,"y 18 == y 21 == y 42 == y 45 == | | | 130 x ?",y=3,u=3,v=3)
mx
}The result matrix looks like this:0 0 0 0 0 0 0 0
0 25 50 55 55 50 25 0
0 50 130 200 200 130 50 0
0 55 200 255 255 200 55 0
0 55 200 255 255 200 55 0
0 50 130 200 200 130 50 0
0 25 50 55 55 50 25 0
0 0 0 0 0 0 0 0

When you add this matrix fourtimes shifted with vectors <0;0>,<0;4>,<4;0>,<4;4>, you get clip with value 255 everywhere. Therefore, this matrix is sufficient for density=2 which means 2x2 shifts per 8x8 block.
It is necessary to write down the average or nominal value of the matrix. This time it is 255.0/4 (because we added 4 shifted and the result was 255) Later in the script it is used for normalizing the merged clips.The value should be stored in WeightMatrixSum function.

My observation is that when the matrix is steep (hi values in the middle and low at hte border) it produces sharper image but also some artefacts in hi detail areas on hi quants (can look like dots). Flat matrices do not make dots, but make edges .

redfordxx

27th May 2006, 04:54

I think I am going to replace the first SmoothDeblockLayer(quant=6 or so...) with quant=3? on half resolution.
Can anybody recommend me some nice downsize and upsize to achieve nice, smooth image after upsizing.

foxyshadis

27th May 2006, 06:57

GaussResize with a low p will probably work, or Bicubic playing with the b and c parameters. (I don't know their behavior well enough to guess what would make for a very blurry resize.)

redfordxx

28th May 2006, 14:40

- it's mod 16 (but prefers 32)
- a little speedup
- added SmoothDeblock2 which has different layers masking, although, I am not sure which can be better...

redfordxx

28th May 2006, 14:47

I must say I like the results of deblocking of the SmoothDeblockLayer function provided it is on correct quant. It restores the detail nicely on places I never thought it is possible.

ITOH I am not satisfied with the layers merging mask. There are always problems al least in the areas, where blocking neighbours detail.

Does anybody have an idea, how to implement the adaptive switching among the layers? --- How to create the masks?

redfordxx

30th May 2006, 01:49

I made some samples of deblocking.
I found six different areas throughout the clip, deblocked, cropped and stacked into one picture.

There are to compare:
oroginal
http://img124.imageshack.us/img124/888/corig8ln.th.png (http://img124.imageshack.us/img124/888/corig8ln.png)
Deblock default
http://img50.imageshack.us/img50/9779/cdeblock2gr.th.png (http://img50.imageshack.us/img50/9779/cdeblock2gr.png)
Deblock stronger
http://img50.imageshack.us/img50/2218/cdeblockmax0qt.th.png (http://img50.imageshack.us/img50/2218/cdeblockmax0qt.png)
DGDecode deblocker default
http://img50.imageshack.us/img50/8023/cdgd2dz.th.png (http://img50.imageshack.us/img50/8023/cdgd2dz.png)
DGDecode deblocker stronger
http://img112.imageshack.us/img112/7499/cdgdmax3co.th.png (http://img112.imageshack.us/img112/7499/cdgdmax3co.png)
SmoothDeblock2(4)
http://img112.imageshack.us/img112/8233/csdb5hg.th.png (http://img112.imageshack.us/img112/8233/csdb5hg.png)
IMO:
SmoothDeblock2 - the deblocking looks very nice and it's most succesful in deblocking. Detail retention is quite good, as long as the detail contrast is over certain level --- otherwise it just disappears (see the lines in the sixth box or drops or thin hair on third box). I hope some tweaking will help. Also I think the total image look is the most similar to original (when fast switching the picture with original, the blocking just disappears and there is no color shift)
Deblock defaults - good detail retention but weak
Deblock stronger - almost as strong as SmoothDeblock and retains more fine detail (means a detail with low contrast), but again (I repeat myself) the hor/vert grid makes the setting higher than default unusable. If there is good detail retention, grid destroys it. You can see it on diagonal lines...
DGDecode (not BlindPP but directly the adaptive one) - a little more succesful in deblocking, than Deblock. The strong setting is already smearing too much.
Now I see, that the adaptivity of DGDecode deblocker suffers the similar problem I complained few posts ago. When meeting hi quant and lo quant blocks, the blocking remains.
(btw, what happened to the fourth box in the strong version? it is quite different from orig)

Conclusion, SmoothDeblock needs
* more detail retention,
* better adaptivity allocation
* probably some protection of "what happens when there is a real edge directly on blockborder"
Then I think it can be OK;-) I don't give up yet...

[EDIT] Newer, more succesful samples later in this thread

krieger2005

30th May 2006, 02:01

Thanks redfordxx. I use your function for an anime. I get the best results with it. I also this "SmoothDeblock2" give the best result (i use it for my encode ;)).
I used your function in combination with one of mine, which i made on the fly. It recover some Detail. Still i encode an anime. But maybe it is usefull... so here is it:

CLmedian(last,SmoothDeblock2(),2,1,1)

function ClMedian(clip c, clip n, int rad, int th1, int th2){
s=gaussianblur(c,rad,rad)
str1="x y - abs "+string(th1)+" > 255 0 ?"
m=YV12LutXY(c,s,str1,str1,str1,Y=3,U=3,V=3)

str2="x y - abs "+string(th2)+" > 255 0 ?"
r=YV12LutXY(c,n,str2,str2,str2,Y=3,V=3,U=3)
r=MaskedMerge(c,n,r,Y=3,V=3,U=3)

MaskedMerge(c,r,m,Y=3,U=3,V=3)
}

Try to play with the red number.

Thanks again for your energy.

708145

2nd June 2006, 09:11

Remember SmoothD? I tried to figure out how it works and made similar script (that§s why I borrowed the name) but adaptive... and slow

[...] shifted DCTFilter [...] weighting mask [...]

I just had a very quick look at your description and code. The basic idea is OK but I think you shouldn't use DCTfilter.
I experimented with DCTfilter as well a while ago and came to the conclusion that it overfilters crisp details.
The alternative I suggest is a filter (DQI.dll) which sits on my HD for almost a year in a preprealpha state :p

A realtime version is possible but my first (all C) implementation is quite slow (and still buggy as hell) :(

But I like your script and I guess it's just a simple substitute of DCTfilter to DQI.

Then, my assumption (not 100% correct) was, blocking is where is hi quantization, so, let's detect the quantization. I detect the quantization with DCTFilter again on certain quant again. That's function QuantDiff. It produces the mask of "what could be removed with lowpass on certain quant".

I have to think more about that idea. At first it sounds great but at second thought I have some doubts.
I fear it will smooth noisy flat regions too much... any test results regarding this?

To sum it up, my plan for a realtime version of this is:
o filter strength detection
o just use 4 shifts
o use DQI with MMX/SSE2
o use a weighting mask

And here is the acronym :D
CAPS - Content adaptive detail preserving smoother

bis besser,
T0B1A5

redfordxx

2nd June 2006, 10:05

###############################################
# SmoothDeblock 0.3 by Redford
#
#
# issues and todo's:
# find best averaging weight matrices for combining DCT shifts (=minimize artefacts at high quants and increase sharpness)
# find best smoothing of the adaptation mask clip or better, adaptationmask shouldnot be block oriented, but only detail oriented
#
# tuning and speed up
#
###############################################

function ShiftDCTBack(clip orig,int density, clip wmc, float wms, string dct_type, int i, int j, int DCTorig, int "quant")
{
l=orig.crop(0,0,Int(8/density)*i+16,-0).FlipHorizontal()
r=orig.crop(orig.Width-Int(8/density)*i-16,0,-0,-0).FlipHorizontal()
cMain=(i==0) ? orig : StackHorizontal(l,orig,r)
t=cMain.crop(0,0,0,Int(8/density)*j+16).FlipVertical()
b=cMain.crop(0,orig.height-Int(8/density)*j-16,0,-0).FlipVertical()
cMain=(j==0) ? cMain : StackVertical(t,cMain,b)
cMain=(i+j==0) ? orig : MergeChroma(cMain,cMain.AddBorders(Int(8/density)*i,Int(8/density)*j,0,0).Crop(0,0,cMain.Width,cMain.Height))
cMain=(i==0) ? cMain : cMain.crop(16,0,orig.Width+16,0)
cMain=(j==0) ? cMain : cMain.crop(0,16,0,orig.Height+16)
cMain=((i+j==0)&&(DCTorig==0)) ? orig : (dct_type=="DCTFilterD") ? cMain.DCTFilterD(quant) : cMain.DCTFilter(DctParams7,DctParams6,DctParams5,DctParams4,DctParams3,DctParams2,DctParams1,DctParams0)
cFlot=mt_lutxy(cMain,wmc,"x y * 255 %",y=3,u=3,v=3)
cMain=mt_lutxy(cMain,wmc,"x y * x y * 255 % - 255 /",y=3,u=3,v=3)
cFlot=(i+j==0) ? cFlot : MergeChroma(cFlot,cFlot.AddBorders(0,0,Int(8/density)*i,Int(8/density)*j).Crop(Int(8/density)*i,Int(8/density)*j,cFlot.Width,cFlot.Height))
cMain=(i+j==0) ? cMain : MergeChroma(cMain,cMain.AddBorders(0,0,Int(8/density)*i,Int(8/density)*j).Crop(Int(8/density)*i,Int(8/density)*j,cMain.Width,cMain.Height))
shiftClip.Set(i*density+j+Int(Pow(density,2)),(i+j==0) ? cFlot : cFlot.crop(Int(8/density)*i,Int(8/density)*j,orig.Width(),orig.height))
shiftClip.Set(i*density+j, (i+j==0) ? cMain : cMain.crop(Int(8/density)*i,Int(8/density)*j,orig.Width(),orig.height))
global avgI = avgI+", shiftClip"+String(i*density+j)+", 255.0/"+String(wms)
global avgF = (i+j==0) ? avgF : avgF+", shiftClip"+String(i*density+j+Int(Pow(density,2)))+", "+String(1/Pow(density,2))
}

function SmoothDeblockLayer(clip orig,float quant,int density, clip wmc, float wms, string "dct_type",int "DCTorig")
{
global gDCTtype=default(dct_type,"DCTFilter") #fixed now
global gDensity=density #number of DCT per 8 pix
global gOrig=orig
global rep=wms*Pow(density,2)
global q=Int(quant)
global matrix1=wmc
global avgI="clipAverageFloat, "+String(1/wms)
global avgF="shiftClip"+String(Int(Pow(density,2)))+", "+String(1/Pow(density,2))
global DCTO=default(DCTorig,0)

Dim("DctParams",8,0)
Dim("shiftClip",Int(Pow(density,2))*2,Blackness())
For2("i=7","i>="+String(quant),"i=i-1",
\"""
DctParams.Set(i,1)
""")
For2("i=0", "i<gDensity", "i=i+1",
\"""
For2("j=0", "j<gDensity", "j=j+1",
\"
ShiftDCTBack(gOrig,gDensity,matrix1,rep,gDCTtype,i,j,DCTO,q)
")
""")
Eval("clipAverageFloat=Average("+avgF+")")
Eval("Average("+avgI+")")
#Subtitle(avgI)
}

function QuantDiff(clip orig,float "quant",string "dct_type")
{
dct_type = default(dct_type,"DCTFilter") #fixed now
quant =default(quant,4)

Dim("DctParams",8,0)
For2("i=7","i>="+String(quant),"i=i-1",
\"""
DctParams.Set(i,1)
""")
#a=AverageBlock8(orig)
a=orig.DCTFilter(1,0,0,0,0,0,0,0)
n=mt_makediff(orig,a,y=3,u=3,v=3)
d=n.DCTFilter(1,1-DctParams6,1-DctParams5,1-DctParams4,1-DctParams3,1-DctParams2,1-DctParams1,1-DctParams0)
d
}

function AdaptionMask(clip orig,int "pregain",int "pregainoffset",int "gain",string "adaption_blursmall",string "adaption_blurbig")
{
pregainoffset =default(pregainoffset,-2)
pregain =default(pregain,100)
gain =default(gain,100)
adaption_blursmall =default(adaption_blursmall,"1")
#adaption_blurbig =default(adaption_blurbig,"1 1 2 2 2 1 1")
adaption_blurbig =default(adaption_blurbig,"1")

o1=orig.mt_lut("x 128 - abs x 128 == 0 "+string(pregainoffset)+" ? + "+string(pregain)+" *",y=3,u=3,v=3)
o1=o1.mt_convolution("0 0 0 0 0 0 0 1 1 1 1 1 1 1 1","0 0 0 0 0 0 0 1 1 1 1 1 1 1 1",y=3,u=3,v=3).AddBorders(0,0,(32 - (orig.width % 32)) % 32,(32 - (orig.height % 32)) % 32)
o1=o1.pointresize(o1.width/8,o1.height/8)
o1=(adaption_blursmall=="1") ? o1 : \
(adaption_blursmall=="expand") ? o1.mt_expand(y=3,u=3,v=3) : \
(adaption_blursmall=="inpand") ? o1.mt_inpand(y=3,u=3,v=3) : \
(adaption_blursmall=="inflate") ? o1.mt_inflate(y=3,u=3,v=3) : \
(adaption_blursmall=="deflate") ? o1.mt_deflate(y=3,u=3,v=3) : \
o1.mt_convolution(adaption_blursmall,adaption_blursmall,y=3,u=3,v=3)
o1=o1.pointresize(o1.width*8,o1.height*8).Crop(0,0,orig.width,orig.height)
o1=(adaption_blurbig=="1") ? o1 : o1.mt_convolution(adaption_blurbig,adaption_blurbig,y=3,u=3,v=3)
o1=o1.mt_lut("x "+string(gain)+" *",y=3,u=3,v=3)
return(o1)
}

function AdaptionMaskDiff(clip orig, clip deb,float "quant", int "pregain", int "pregainoffset", int "gain", int "thresh", string "adaption_blur", int "borders", int "expands" ,string "dct_type")
{
pregainoffset =default(pregainoffset,-3)
pregain =default(pregain,100)
borders =default(borders,0)
expands =default(expands,1)
gain =default(gain,8)
thresh =default(thresh,2)
adaption_blur =default(adaption_blur,"1 2 1")
dct_type =default(dct_type,"DCTFilter") #fixed now
quant =default(quant,4)

Dim("DctParams",8,0)
For2("i=7","i>="+String(quant),"i=i-1",
\"""
DctParams.Set(i,1)
""")
#a=AverageBlock8(orig)
a=orig.DCTFilter(1,0,0,0,0,0,0,0)
n=mt_makediff(orig,a,y=3,u=3,v=3)
d=n.DCTFilter(1,1-DctParams6,1-DctParams5,1-DctParams4,1-DctParams3,1-DctParams2,1-DctParams1,1-DctParams0)

o1=d.mt_lut("x 128 - abs x 128 == 0 "+string(pregainoffset)+" ? + "+string(pregain)+" *",y=3,u=3,v=3)
o1=o1.DCTFilter(1,0,0,0,0,0,0,0)
o1=o1.mt_lut("x "+string(pregain/5)+" *",y=3,u=3,v=3)

diff=mt_makediff(deb,orig,y=3,u=3,v=3)
diff2=mt_lutxy(o1,diff,"y 128 - x * 255 / abs "+string(thresh)+" -",y=3,u=3,v=3)
p1=diff2.PtrnBlock(3,borders,255,255,255)
diff3=mt_lutxy(diff2,p1,"x y * 255 / "+string(gain)+" *",y=3,u=3,v=3)
diff4=diff3.mt_expand(y=3,u=3,v=3)
(adaption_blur=="1") ? diff4 : diff4.mt_convolution(adaption_blur,adaption_blur,y=3,u=3,v=3)
mt_lut("x "+string(gain)+" *",y=3,u=3,v=3)
(expands>0) ? mt_expand(y=3,u=3,v=3) : last
(expands>1) ? mt_expand(y=3,u=3,v=3) : last
(expands>2) ? mt_expand(y=3,u=3,v=3) : last
(expands>3) ? mt_expand(y=3,u=3,v=3) : last
(expands>4) ? mt_expand(y=3,u=3,v=3) : last
}

function WeightMatrixClip(clip orig, string "name",int "level")
{
level=default(level,0)
clp=orig.Crop(0,0,16,16).Trim(1,1)
name=default(name,"04center255x2")

mx1 = (name=="04all78")||(name=="04center255x2")||(name=="16center255") ? clp : clp.Matrix(3).PtrnFormat(clp)
mx = (name=="04center255border0smooth_1") ? clp.PtrnBlock(3,0,50,200,255) : \
(name=="16center255border0smooth_1") ? clp.PtrnBlock(3,0,70,185,255) : \
(name=="16center255border0smooth_2") ? clp.PtrnBlock(3,100,150,220,255) : clp
#middle1
mx = (name=="04all78")||(name=="04center255x2")||(name=="16center255") ? mx : mt_lutxy(mx,mx1,"y 12 == y 11 == y 25 == y 30 == y 33 == y 38 == y 51 == y 52 == | | | | | | | " + ( \
(name=="04center255border0smooth_1") ? "55" : \
(name=="16center255border0smooth_1") ? "115" : \
(name=="16center255border0smooth_2") ? "180" : \
"x") + " x ?",y=3,u=3,v=3)
#diag0
#diag1
mx = (name=="04all78")||(name=="04center255x2")||(name=="16center255") ? mx : mt_lutxy(mx,mx1,"y 9 == y 14 == y 54 == y 49 == | | | " + ( \
(name=="04center255border0smooth_1") ? "25" : \
(name=="16center255border0smooth_1") ? "20" : \
(name=="16center255border0smooth_2") ? "130" : \
"x") + " x ?",y=3,u=3,v=3)
#diag2
mx = (name=="04all78")||(name=="04center255x2")||(name=="16center255") ? mx : mt_lutxy(mx,mx1,"y 18 == y 21 == y 42 == y 45 == | | | " + ( \
(name=="04center255border0smooth_1") ? "130" : \
(name=="16center255border0smooth_1") ? "140" : \
(name=="16center255border0smooth_2") ? "200" : \
"x") + " x ?",y=3,u=3,v=3)

(name=="16center255") ? (clp.PtrnBlock(3,0,0,0,255)) : \
(name=="04center255x2") ? (clp.PtrnBlock(3,0,0,255,255)): \
(name=="04all78") ? (clp.PtrnBlock(3,150,150,212,212)): \
mx
#mt_lut(String(level)+" 255 "+String(level)+" - 255 / x * +",y=3,u=3,v=3)
PtrnFormat(orig)
}

function WeightMatrixSum(string "name",int "level")
{
level=default(level,0)
l=(name=="16center255") ? (255.0/16) : \
(name=="04center255x2") ? (255.0/4): \
(name=="04center255border0smooth_1") ? (255.0/4): \
(name=="16center255border0smooth_1") ? (1155.0/16): \
(name=="16center255border0smooth_2") ? (2385.0/16): \
(name=="04all78") ? (662.0/4): \
(255.0/4)
#return((255.0-level)/255.0*l+level)
l
}

function BorderMask(clip orig, int "bth1", int "bth2", int "bshare", int "eth1", int "eth2", int "eshare", float "buvmod", float "euvmod")
{
bth1=default(bth1,0)
bth2=default(bth2,2)
bshare=default(bshare,200)
eth1=default(eth1,30)
eth2=default(eth2,50)
eshare=default(eshare,0) #0=real edge protection off
buvmod=default(buvmod,2.0)
euvmod=default(euvmod,2.4)

ph=orig.PtrnHorizontal(3,0,1,2,3,4,5,6,7)
pv=orig.PtrnVertical(3,0,1,2,3,4,5,6,7)
h1a=orig.mt_lutxy(ph,"y 0 == y 7 == | x 128 ?",y=3,u=3,v=3)
v1a=orig.mt_lutxy(pv,"y 0 == y 7 == | x 128 ?",y=3,u=3,v=3)
h1b=orig.mt_lutxy(ph,"y 1 == y 6 == | x 128 ?",y=3,u=3,v=3)
v1b=orig.mt_lutxy(pv,"y 1 == y 6 == | x 128 ?",y=3,u=3,v=3)
h2a=h1a.mt_convolution(horizontal="1",vertical="1 -1 1",total=1,y=3,u=3,v=3)
v2a=v1a.mt_convolution(horizontal="1 -1 1",vertical="1",total=1,y=3,u=3,v=3)
h2b=h1b.mt_convolution(horizontal="1",vertical="1 -1 1 -1 1",total=1,y=3,u=3,v=3)
v2b=v1b.mt_convolution(horizontal="1 -1 1 -1 1",vertical="1",total=1,y=3,u=3,v=3)
v2=mt_lutxy(v2a,v2b,"x 128 - y 128 - 3 / - abs x 128 - abs > x x y 128 - 3 / - ?",y=3,u=3,v=3)
h2=mt_lutxy(h2a,h2b,"x 128 - y 128 - 3 / - abs x 128 - abs > x x y 128 - 3 / - ?",y=3,u=3,v=3)
v3b=v2.mt_lutxy(pv,"y 0 == y 7 == | x 128 ? 128 - abs 5 * 1.1 * ",y=3,u=3,v=3).mt_convolution(horizontal="1",vertical="1 1 1",y=3,u=3,v=3)
h3b=h2.mt_lutxy(ph,"y 0 == y 7 == | x 128 ? 128 - abs 5 * 1.1 * ",y=3,u=3,v=3).mt_convolution(horizontal="1 1 1",vertical="1",y=3,u=3,v=3)
v3e0=v2a.mt_convolution(horizontal="1",vertical="1 1 1 1 1 1 1 1 1",y=3,u=2,v=2).mt_convolution(horizontal="1",vertical="1 1 1 1 1 1 1",y=2,u=3,v=3)
h3e0=h2a.mt_convolution(horizontal="1 1 1 1 1 1 1 1 1",vertical="1",y=3,u=2,v=2).mt_convolution(horizontal="1 1 1 1 1 1 1",vertical="1",y=2,u=3,v=3)
v3e=(eshare==0) ? orig : v3e0.mt_lutxy(pv,yexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1)+" - "+String(eth2)+" "+String(eth1)+" - / 255 *", \
uexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1/euvmod)+" - "+String(eth2/euvmod)+" "+String(eth1/euvmod)+" - / 255 *", \
vexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1/euvmod)+" - "+String(eth2/euvmod)+" "+String(eth1/euvmod)+" - / 255 *",y=3,u=3,v=3)
h3e=(eshare==0) ? orig : h3e0.mt_lutxy(ph,yexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1)+" - "+String(eth2)+" "+String(eth1)+" - / 255 *", \
uexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1/euvmod)+" - "+String(eth2/euvmod)+" "+String(eth1/euvmod)+" - / 255 *", \
vexpr="y 0 == y 7 == | x 128 ? 128 - abs "+String(eth1/euvmod)+" - "+String(eth2/euvmod)+" "+String(eth1/euvmod)+" - / 255 *",y=3,u=3,v=3)
v4e=(eshare==0) ? orig : v3e.HorizontalLinearBlock
h4e=(eshare==0) ? orig : h3e.VerticalLinearBlock
e1=(eshare==0) ? orig : mt_lutxy(v4e,h4e,"x y > x y ?",y=3,u=3,v=3)
e2=(eshare==0) ? orig : e1.mt_lut("255 x - "+String(eshare)+" * 255 / "+String(eshare)+" - 255 +",y=3,u=3,v=3)
v4b=(eshare==0) ? v3b : v3b.mt_lutxy(v3e,"x y 1 + /",y=3,u=3,v=3)
h4b=(eshare==0) ? h3b : h3b.mt_lutxy(h3e,"x y 1 + /",y=3,u=3,v=3)
v5b=v4b.HorizontalLinearBlock
h5b=h4b.VerticalLinearBlock
b1=mt_lutxy(v5b,h5b,yexpr="x y > x y ? 5 / "+String(bth1)+" - "+String(bth2)+" "+String(bth1)+" - / 255 *", \
uexpr="x y > x y ? 5 / "+String(bth1/buvmod)+" - "+String(bth2/buvmod)+" "+String(bth1/buvmod)+" - / 255 *", \
vexpr="x y > x y ? 5 / "+String(bth1/buvmod)+" - "+String(bth2/buvmod)+" "+String(bth1/buvmod)+" - / 255 *",y=3,u=3,v=3)
b2=b1.mt_convolution(horizontal="1 1 1 1 1",vertical="1 1 1 1 1",y=3,u=3,v=3)
b3=b2.mt_lut("x "+String(bshare)+" * 255 / "+String(bshare)+" - 255 +",y=3,u=3,v=3)
(eshare==0) ? b3 : mt_lutxy(e2,b3,"x y < x y ?",y=3,u=3,v=3)
}

redfordxx

2nd June 2006, 10:07

redfordxx

2nd June 2006, 11:31

I suggest is a filter (DQI.dll)
And what is the difference?

I have to think more about that idea. At first it sounds great but at second thought I have some doubts.
I fear it will smooth noisy flat regions too much... any test results regarding this?

Well the question is whether smoothing noisy areas is bad or good;-) But definitely, fine detail was problem in SmoothDeblock2. SmoothDeblock3 is little better... But it does not remove the cause of the problem it only rescues something of what was damaged

To sum it up, my plan for a realtime version of this is:

I always thought that coding this script would dramatically increase speed, because is the script many operations are useless just because I have to apply the operation on whole clip. But the idea still needs to be improved.

o just use 4 shifts

benefits of 16:
little sharper and less artifact (artifacts appear when are highly quantized hi freq areas... compare SmoothDeblockLayer(quant=6,density=2 or 4))
you can choose any weighting mask you want

And here is the acronym :D
CAPS - Content adaptive detail preserving smoother
Hmmm shouldn't be the word deblock in the name? (reason - search)

I have one more approach to try to implement: First adapting and then merging shifts.
For instance split the frame to areas 32x32 and apply appropriate quant only to that area. It would be milions of crops in script...;-)
For this is needed very good and smooth adaptation mask prepared.

708145

2nd June 2006, 15:23

DQI is a simple filter that simulates real quantization similar to jpeg compression and decompression.

Well the question is whether smoothing noisy areas is bad or good;-) But definitely, fine detail was problem in SmoothDeblock2. SmoothDeblock3 is little better... But it does not remove the cause of the problem it only rescues something of what was damaged

benefits of 16:
little sharper and less artifact (artifacts appear when are highly quantized hi freq areas... compare SmoothDeblockLayer(quant=6,density=2 or 4))

So you are curing the symptoms ;)
Maybe improving the previous step is better.

I have one more approach to try to implement: First adapting and then merging shifts.
For instance split the frame to areas 32x32 and apply appropriate quant only to that area. It would be milions of crops in script...;-)
For this is needed very good and smooth adaptation mask prepared.

Yeah! As a script it's ugly. As a filter even 8x8 is easy and fast.

bis besser,
T0B1A5

redfordxx

2nd June 2006, 18:19

So you are curing the symptoms ;)
Maybe improving the previous step is better.
100% agree... but for now... it was just faster.

But the "real edge protection"...I have no idea at the moment, how to improve it:(

Yeah! As a script it's ugly. As a filter even 8x8 is easy and fast.Agree again...
... but I wanted first to test it a little before "convincing someone else that it is worth coding".

OTOH, if this would be coded not as one filter but as a set of filters, the partial solution can be made.

Let's say the final solution would be:
1 - prepare adapt masks
2 - shift clips
3 - adaptively apply DCT on shifts
4 - shift back
5 - mask merge

Then for step 3 could be 1 filter (name DCTFilterMask) with two input clips:
- original
- 8xreduced resolution mask
DCTFilter(c,i0,i1,i2,i3,i4,i5,i6,i7) filter normally applies i0-i7 parameters
DCTFilterMask(c,m) would read the parameters for each block in the pixel of clip m. I wonder, whether the DCTFilter can be modified that way easily
or your filter:
DQIFilterMask(c,m,pathtoquantmatrixtxtfile) - m would contain quantizers. I know you mentioned somewhere you use 6of9 as default, but from many sources it is possibler to read the true quant matrix, so why not to use it.

steps 5 can be maybe easily solved by modifying the Average filter... look here (http://forum.doom9.org/showthread.php?p=829568#post829568)

with these two filters the script would be much faster and there is more time for experimenting with the adapt mask creation
and later, after it is tested, the adapt matrix creation filter will be made

foxyshadis

2nd June 2006, 22:48

I think the real edge detection could use tweaking; I'm not exactly sure how it works now, but real edges are almost never perfectly horizontal/vertical for 8 pixels. On the other hand, very bad sources routinely have huge differences between block edges where they were way overcompressed. Perhaps testing by threshold, and if it fails that, test compared to the block's average color (or an average of x nearby pixels within the block or something, get as complex as you want), if it's close enough it's probably a block anyway.

In fact, if you could pick up a rating of the block's overall complexity, you don't need a nearby color test at all, if it exceeds the threshold but has practically no detail, deblock the hell out of it. If both have lots of detail but exceed the threshold, then reduce. If just one does, try pushing the contents of the high-detail block into the low-detail one.

A filter that can color every pixel in an 8x8 block as a rating of its high frequency content would be perfect for measuring via script operations. It would be an excellent way of estimating initial blocking as well. (Unless you have some awesome way of gathering that information from DCTfilter's output that I don't see.)

redfordxx

2nd June 2006, 23:54

Starting from end:
A filter that can color every pixel in an 8x8 block as a rating of its high frequency content would be perfect for measuring via script operations. (Unless you have some awesome way of gathering that information from DCTfilter's output that I don't see.)
For SmoothDeblock1 it is done by QuantDiff + AdaptionMask. Example (try and see)

c1=orig.QuantDiff(quant=5) #returns something like DCTFilter(0,0,0,1,1,1,1,1)+128... so it is highpass
c2=c1.AdaptionMask(pregain,pregainoffset,gain,b1,b2) #makes some crazy averaging and blurring etc. as follows
#c1=abs(c1-128)
#if c1>0 then c1=c1+pregainoffset
#c1=c1*pregain ... increasing contrast
#average each block into one pixel (=convolution+pointresize)
#according to adaption_blursmall there is some blurring (see the script)
#pointresize to original size
#again blurring according to adaption_blurbig (see the script)
#c2=last*gain ... increasing contrast again
So the result should be this 8x8 block rating blurred into each other

For SmoothDeblock2 it is integrated into one function --- AdaptionMaskDiff --- and I am trying to follow more the detail than only the blocks

But to decide properly what to do with that info is not easy for me. Example:
on block1 QuantDiff(6) returns small value.
on block2 QuantDiff(6) returns bigger value.
But..
QuantDiff(2) returns the same value on block1.
QuantDiff(2) returns zero on block2.
It means that
block2 has lot of mid frequency detail and none hi freq.
block1 has little detail but on hi freq

What should be filtered stronger? Maybe there is something about DCT I am missing....

I can have for each block 8 x info about amount of frequency (for each quant) and probably best would be to map it on <0;255>, so that I can say
if there is 128 let's use SmoothDeblockLayer(quant=4)
if there is 255 let's use original
etc. The values would be continuous, so there can be merging of two neighbouring layers.

It would be an excellent way of estimating initial blocking as well.I use it to detect detail so that I know how much smoothing I can afford without destroying the detail, in fact here I don't care whether there is blocking or not (this I tried to improve in version 3)

How would you estimate the blocking?

redfordxx

3rd June 2006, 01:08

Improved SmoothDeblock3:

- luma and chroma have different thresholds in BorderMask (the difference can be set by buvmod, resp. euvmod)

- Wow, I relized, that
mt_lutxy(..., yexpr=a, uexpr=b, vexpr=c,y=3,u=3,v=3)
is different than
mt_lutxy(..., a, b, c,y=3,u=3,v=3)
surprises all the time...

redfordxx

3rd June 2006, 01:22

Original
http://img64.imageshack.us/img64/1879/orig5yj.th.png (http://img64.imageshack.us/img64/1879/orig5yj.png)
Smoothdeblock1
http://img173.imageshack.us/img173/9480/sdb13pb.th.png (http://img173.imageshack.us/img173/9480/sdb13pb.png)
Smoothdeblock2
http://img173.imageshack.us/img173/4899/sdb27ge.th.png (http://img173.imageshack.us/img173/4899/sdb27ge.png)
Smoothdeblock3
http://img173.imageshack.us/img173/8733/sdb38ic.th.png (http://img173.imageshack.us/img173/8733/sdb38ic.png)

Hm, 2-->3 is visible quality improvement...
but 1-->2 seems to get worse...weird

maybe it'l be worth trying to apply BorderMask on SmoothDeblock1...

redfordxx

3rd June 2006, 02:16

I think the real edge detection could use tweaking; I'm not exactly sure how it works now, but real edges are almost never perfectly horizontal/vertical for 8 pixels. On the other hand, very bad sources routinely have huge differences between block edges where they were way overcompressed. Perhaps testing by threshold, and if it fails that, test compared to the block's average color (or an average of x nearby pixels within the block or something, get as complex as you want), if it's close enough it's probably a block anyway.

Hmm, true...
All the result inside the picture was that few edges are not deblocked and the artificial edges (where the pictures are joined) are still blurred somewhere... so so maybe it is really better for now to let the probability do the job;)

If just one does, try pushing the contents of the high-detail block into the low-detail one.
I see one trick how to do it maybe:
1.deblock
2.adaptively merge with original
(now the hi detail blocks are original again, but the lo detail blocks are half way to the hi detail neighbours)
3.deblock again

But I don't know, maybe it can cost lot of detail. And would be pretty slow.This one is easy but there are faster ways probably.

redfordxx

3rd June 2006, 11:53

Hi guys,

posting again, asking for opinions. How to quantize areas where two different complexity blocks meet:
(again exemple with numbers=colors... you can draw it in a graph on the paper to better visualize)

in first column are blocks to be smoothed, the shift for DCTFilter is marked underlined
then follow estimated quants
In 2D I have 2 or 4 quants for each shifted block and I have to make one from it, to apply DCT filter. There are options average or minimum (or something between).
Finally there is an example what could DCTFilter do at minimum quant and avg quant.

block quant smoothed underlined part
blockA blockB quantA quantB --> avgquant, minquant avgquant minquant
20 40 20 40 20 40 20 40|10 10 10 10 10 10 10 10 1 7 5 1 25 30 25 20 15 12 10 10 20 38 12 10 10 10 10 10
----------------------- (bad) (ok)

20 40 20 40 20 40 20 20|10 10 10 10 10 10 10 10 1 7 5 1 20 16 13 12 11 10 10 10 20 19 11 10 10 10 10 10
----------------------- (ok) (bad)

20 40 40 40 40 40 40 40|20 20 20 20 20 20 20 40 1 1 1 1 40 38 22 20 20 20 20 20 40 38 22 20 20 20 20 20
----------------------- (ok) (ok)

If it is ever possible to follow the detail with the quant rating...

per pixel quant smoothed underlined part
blockA blockB A B --> avgquant, minquant avgquant minquant
20 40 20 40 20 40 20 40|10 10 10 10 10 10 10 10 1111111177777777 5 1 25 30 25 20 15 12 10 10 20 38 12 10 10 10 10 10
----------------------- (bad) (ok)

20 40 20 40 20 40 20 20|10 10 10 10 10 10 10 10 1111126777777777 7 6 19 17 15 14 13 12 11 10 20 17 15 13 12 11 10 10
----------------------- (ok) (ok)

20 40 40 40 40 40 40 40|20 20 20 20 20 20 20 40 1177777777777711 7 7 40 35 30 25 20 15 12 10 40 35 30 25 20 15 12 10
----------------------- (bad) (bad)

So, for each approach there is a situaltion where it fails... so what to choose?
Any opinions?

redfordxx

3rd June 2006, 13:14

Is that correct way how to avoid processing of Filters 2 and 4 based on param avoid? (best for CPU and memory)
c1=source
c2=c1.Filter1()
c3=(avoid==1) ? c2 : c2.Filter2()
c4=c3.Filter3()
c5=(avoid==1) ? c4 : c4.Filter4()
c6=c5.Filter5()

foxyshadis

3rd June 2006, 15:03

Yep. At most it might cause extra frame copies, which aren't that bad.

(Still experimenting with AdaptionMaskDiff and other functions so I can make more informed comments about how the filter works. Brain still processing your last post.)

redfordxx

3rd June 2006, 22:49

Still experimenting with AdaptionMaskDiff and other functions
So, here is new food for your brain. This is start of implementation the new idea I mentioned: First adaptive quantization and only then maskmerging the shifts.

It is done for 7 layers(!) for studying purposes but maybe 7 remain.

This is only one shift. More shifts will be later merged. But setting up this is crucial. I think it is more readable than the scripts before (seems to be long, but it's because everything is 7times there)

Criteria of this working should be:
remove original blocking
retain detail
creating some new blocking is acceptable (because it would be removed by merging)

There are four spots mentioned in the script I see to be improved:
2x tweaking (not yet done at all)
1x blur
1x post#26
#32 needed for reduced resolution only
global modulo=16

#black borders away
crop_l=0
crop_r=4
crop_t=0
crop_b=10

o0=source...

#make it larger (prepare for shifts)
o0h=StackHorizontal(o0.crop(crop_l,0,modulo+crop_l,0).FlipHorizontal(),o0.crop(crop_l,0,-crop_r,0),o0.crop(o0.width-modulo-crop_r*2,0,-crop_r,0).FlipHorizontal())
o0hv=StackVertical(o0h.crop(0,crop_t,0,modulo+crop_t).FlipVertical(),o0h.crop(0,crop_t,0,-crop_b),o0h.crop(0,o0h.height-modulo-crop_b*2,0,-crop_b).FlipVertical())

#get scale (used for all shifts)
dm2=o0hv.QuantScale()

#smooth shift(4,4)
o0hv.AdaptiveDCT(dm2,4,4,modulo,o0.width,o0.height)

#luma only
#mt_lut(y=3,u=-128,v=-128)

function AdaptiveDCT(clip o, clip detailmask, int sh, int sv, int offset, int origwidth, int origheight)
{
#cut out the shifted area
d0l=o.crop(offset-sh,offset-sv, ((origwidth+sh) % modulo == 0) ? (origwidth+sh) : (origwidth+sh)-((origwidth+sh) % modulo)+modulo,((origheight+sv) % modulo) == 0 ? (origheight+sv) : (origheight+sv)-((origheight+sv) % modulo)+modulo)
d0c=o.crop(offset-sh*2,offset-sv*2, ((origwidth+sh) % modulo == 0) ? (origwidth+sh) : (origwidth+sh)-((origwidth+sh) % modulo)+modulo,((origheight+sv) % modulo) == 0 ? (origheight+sv) : (origheight+sv)-((origheight+sv) % modulo)+modulo)
d0=MergeChroma(d0l,d0c)

#define scale - thresholds for different quants
#Maybe here is also a tweaking to be done !
l0=255
l1=219
l2=182
l3=146
l4=109
l5=73
l6=36
l7=0
xfade=0

#shifting the detail mask
#problem of post #26 to solve
m=detailmask.mt_convolution(horizontal=string(sh)+" "+string(8-sh)+" 0",vertical=string(sv)+" "+string(8-sv)+" 0",total=64.0,y=3,u=3,v=3)
#end of problem of post #26 to solve

#creating masks for each quant (map <l0,l1> to <0,255> etc.)
mm=m.crop(offset/8,offset/8,d0.width/8,d0.height/8)
m6=mm.mt_lut("x "+String(l7)+" - "+String(xfade)+" + "+String(l6)+" "+String(l7)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m5=mm.mt_lut("x "+String(l6)+" - "+String(xfade)+" + "+String(l5)+" "+String(l6)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m4=mm.mt_lut("x "+String(l5)+" - "+String(xfade)+" + "+String(l4)+" "+String(l5)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m3=mm.mt_lut("x "+String(l4)+" - "+String(xfade)+" + "+String(l3)+" "+String(l4)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m2=mm.mt_lut("x "+String(l3)+" - "+String(xfade)+" + "+String(l2)+" "+String(l3)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m1=mm.mt_lut("x "+String(l2)+" - "+String(xfade)+" + "+String(l1)+" "+String(l2)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)
m0=mm.mt_lut("x "+String(l1)+" - "+String(xfade)+" + "+String(l0)+" "+String(l1)+" - "+String(xfade)+" 2 * + / 255 *",y=3,u=3,v=3)

#upsize masks
f6=m6.pointresize(mm.width*8,mm.height*8)
f5=m5.pointresize(mm.width*8,mm.height*8)
f4=m4.pointresize(mm.width*8,mm.height*8)
f3=m3.pointresize(mm.width*8,mm.height*8)
f2=m2.pointresize(mm.width*8,mm.height*8)
f1=m1.pointresize(mm.width*8,mm.height*8)
f0=m0.pointresize(mm.width*8,mm.height*8)

#blur masks?
#
#

#apply dct on clips (different quants)
d1=d0.DCTFilter(1,1,1,1,1,1,1,0)
d2=d0.DCTFilter(1,1,1,1,1,1,0,0)
d3=d0.DCTFilter(1,1,1,1,1,0,0,0)
d4=d0.DCTFilter(1,1,1,1,0,0,0,0)
d5=d0.DCTFilter(1,1,1,0,0,0,0,0)
d6=d0.DCTFilter(1,1,0,0,0,0,0,0)
d7=d0.DCTFilter(1,0,0,0,0,0,0,0)

#merge according to masks
d=d7
d=mt_merge(d,d6,f6,y=3,u=3,v=3)
d=mt_merge(d,d5,f5,y=3,u=3,v=3)
d=mt_merge(d,d4,f4,y=3,u=3,v=3)
d=mt_merge(d,d3,f3,y=3,u=3,v=3)
d=mt_merge(d,d2,f2,y=3,u=3,v=3)
d=mt_merge(d,d1,f1,y=3,u=3,v=3)
d=mt_merge(d,d0,f0,y=3,u=3,v=3)

#shift back
a0l=d.crop(sh,sv,origwidth,origheight)
a0c=d.crop(sh*2,sv*2,origwidth,origheight)
a0=MergeChroma(a0l,a0c)
a0
}

function QuantScale(clip clp)
{
#detect high frequencies (for each quant separately)
qm7=clp.QuantDiff(7)
qm6=clp.QuantDiff(6)
qm5=clp.QuantDiff(5)
qm4=clp.QuantDiff(4)
qm3=clp.QuantDiff(3)
qm2=clp.QuantDiff(2)
qm1=clp.QuantDiff(1)

#make abs and increase contrast
am7=qm7.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am6=qm6.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am5=qm5.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am4=qm4.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am3=qm3.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am2=qm2.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)
am1=qm1.mt_lut("x 128 - abs 30 *",y=3,u=3,v=3)

#average
dm7=am7.DCTFilterD(14)
dm6=am6.DCTFilterD(14)
dm5=am5.DCTFilterD(14)
dm4=am4.DCTFilterD(14)
dm3=am3.DCTFilterD(14)
dm2=am2.DCTFilterD(14)
dm1=am1.DCTFilterD(14)

#downsize for higher speed
rs7=dm7.PointResize(clp.width/8,clp.height/8)
rs6=dm6.PointResize(clp.width/8,clp.height/8)
rs5=dm5.PointResize(clp.width/8,clp.height/8)
rs4=dm4.PointResize(clp.width/8,clp.height/8)
rs3=dm3.PointResize(clp.width/8,clp.height/8)
rs2=dm2.PointResize(clp.width/8,clp.height/8)
rs1=dm1.PointResize(clp.width/8,clp.height/8)

#merge into one scale (remember that when all coefficients are 1, the highest frequencies are here 7 times, second highest 6 times etc.)
####### Here is the main tweaking to be done !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Average(rs1,1,rs2,1,rs3,1,rs4,1,rs5,1,rs6,1,rs7,1)
}

Clown shoes

4th June 2006, 02:14

Didée

4th June 2006, 03:20

my source file definatley has multiple of 4 dimensions.
W00t! Where did you get that N-dimensional input file from? Are there any string-theorem based Avisynth Filters out there? :D

For the moment, SmoothDeblock seems to need MOD32 (or MOD16 at least?) input resolutions, as mentioned in the first post. For testing purposes, just add borders if needed. In case of serious use, better apply "padding(x,y)" (provided by MVTools) before, and crop back later on.

@ redford
Nice progress with your mission - looking forward to where the journey goes. :)

The mere deblocking work of SmoothDeblock3 is just great. Biggest concern really is detail retention. (For natural sources in any case, for animated content probably less.)
Let me throw in a point here - can't say if it fits in the current framework, but give a thought on it:

Any "real" source detail is very likely to appear in an anti-aliased form at the pixel level, i.e. the slope of the signal is continuous.
Digital Artefacts, like blocking, are very likely to show uncontinuous pixel transitions. Like

10 10 12 20 20 18 10

is an anti-aliased step, hence is likely to be real signal, where

10 10 10 20 20 10 10 / or even / 10 10 09 21 20 10 10

has a broken slope, hence is likely to be artificial. asfaso.

Examining lots of blocks and at lots of detail, this seems to be true predominantly (however not always). A relatively easy check would be to look out if a pixel has two adjacent spatial neighbors where the gradient spanned by these neighbors is pointing to the value of the actual pixel, or not.

It could be a good idea to include this property into the distinction or weighting between what shall be removed, and what shall be retained.

However, doing these pixel comparisons through scripting is not all that easy. Probably some parts of the existent frequency filters could be of help here.

redfordxx

4th June 2006, 04:25

I am trying to use SmoothDeblock3, but vdub is returning an error when trying to load my script; YV12 width must be multiple of 4. I have Patterns, Arrays and loops, DCTFilter and Average in my plugins folder and my source file definatley has multiple of 4 dimensions. My script is simply;

avisource("E:\test.avi")
SmoothDeblock3()
Hm, I thought I made it 16... so I missed something somewhere.
Anyway, there is downsizing by 8 in the script, so 32/8=4 so mod32 is necessary. I'll fix it later.

Until then, use Didée's suggestion or try the beginning part of the script two posts ago until the line #get scale (used for all shifts). write modulo=32.
It mirrors the borders so that there are not black borders leaking into the picture. You can also crop thin black lines. However it adds more borders than necessary, so you have to either crop or modify the part of script.

First look what it does, play with parameters and after you know what you want, add SmoothDeblock.

redfordxx

4th June 2006, 05:32

1.
@Didée

I see your point... I will come back to this idea at the end

If I see it correctly, it implies that I shouldnot worry about 3rd example in second code-block in post #26
...and it implies that theoretically best would be option minquant per pixel
...and it implies that I should find out how to create not block related but per pixel quant mask (reliable)

am I correct?

-----------------------------------------------------
2.
Oh, I have kinda bug in QuantDiff function... I have to look closer on it again.

------------------------------------------------------
3.
I have another idea how to detect detail
let's have a
mt_makediff(orig.DCTFilter(1,1,0,0,0,0,0,0),orig)

suppose it is
120 120 120 120 152 152 120 120
when I subtract median of the block which is here 120, I receive
128 128 128 128 160 160 128 128
Is there some filter which returns median of an 8x8 block?

However, when there is edge like
120 120 120 126 130 136 136 136
we have problems...:(

Didée

4th June 2006, 06:48

Is there some filter which returns median of an 8x8 block?
Yes. Clouded once made an 8x8-block restricted, variable radius median filter. Look here (http://people.pwf.cam.ac.uk/mg262/posts/) for BlockMedian.

But that doesn't help much, it's not that easy. Detecting detail just by highpassing, be it average, gauss or median, gives no information about what kind of "detail" was detected. (As in denoising: highpassing returns most of the noise, sure, but that alone doesn't help...)
Moreover, BlockMedian with r=8 very often gives "interesting" (unusabe) results, if if they're numerically correct. :)

If I see it correctly, it implies that I shouldnot worry about 3rd example in second code-block in post #26
...and it implies that theoretically best would be option minquant per pixel
...and it implies that I should find out how to create not block related but per pixel quant mask (reliable)

am I correct?
Can't tell, haven't wrapped my mind too much about either this multiple-i/dct approach, or about your small and overseeable ;) script implementation yet.

If there was an easy way of detecting those uncontinuous pixel transitions, I'd try the following:

- smooth uncontinuities:
-- in-block: use this as-is (smooth potential in-block artefacts)
-- block borders: here, this gives the "block border correction"

- IF "block border correction" was done to pixels that have NO hi-frequency in-block neighbors:
-- spread border correction:
--- to the middle of block if the opposite block border is "not blocky"
--- over the whole block's width if opposite block border is "blocky", too
- ELSE:
-- do not spread border correction, just smooth the border

Not sure how far this aligns with the shifted i/dct approach. However, that is the path I have in front of my inner eye ... in the end, the goal is to remove those artefacts that we do *see*, isn't it. And if so, thinking about what is the special property of those artefacts that make them so outstanding to the human eye, in contrary to "real detail", seems a worthwhile strategy ... ;)

Soulhunter

4th June 2006, 10:24

W00t! Where did you get that N-dimensional input file from? Cube II (http://en.wikipedia.org/wiki/Cube_2:_Hypercube) maybe... XD

Bye

redfordxx

4th June 2006, 12:18

I am playing with this function for detecting detail:function EdgeDetect(clip clp)
{
qm=clp
qm1=qm.PointResize(qm.width/8*9,qm.height/8*9)
qm2=qm1.mt_convolution("1 0 0","1 0 0",y=3,u=3,v=3)
qm3=qm2.PointResize(qm.width/8*10,qm.height/8*10)
minmax=mt_luts(qm3,qm3, mode = "max", pixels = mt_square( 1 ), expr = "x y - abs 20 *",y=3,u=3,v=3)
qm4=minmax.mt_convolution("0 0 0 0 0 0 1","0 0 0 0 0 0 1",y=3,u=3,v=3)
#qm4=qm4.crop(2,2,0,0).AddBorders(0,0,2,2)# will be faster? but take care of chroma
qm5=qm4.PointResize(qm2.width,qm2.height)
#qm5=qm5.mt_convolution("0 0 1","0 0 1",y=3,u=3,v=3)# will be faster? but take care of chroma
qm6=qm5.PointResize(qm.width,qm.height)
qm6
}

It is just maximum difference (call it md later in the text) to neighbour pixels calculated by mt_luts. Set of resizers is there to avoid border edges. There are still some border relicts visible, but that not my point now.

Here I see the light at the end of the tunnel and I need your help here, because I forgot all of these fourier and transform formulas I learned at university:( :

My guess is, there should be relatively exact mathematic relation between md value and strenght of quantization.

Relation saying something like:
if md>20 then DCTFilter(1,1,1,1,0,0,0,0) would smear it
if md>30 then DCTFilter(1,1,1,1,1,0,0,0) would smear it
etc.

Oh, maybe not. Maybe we need something like second derivation... yes, more probably, definitely. So double application of maximum difference is necessary and then apply formula to map these values onto (0;255), where 0=maximum quantization (only first coefficient remains) and 255=no quantization (original clip should be used).
We could obtain quite detail quantization mask, if lucky...

Pls, can anybody take a look at it?

[EDIT]I see now, the scond derivation can't be calculated like that, but I believe it is possible

redfordxx

5th June 2006, 11:28

Again I am feeding my onw thread.

I want to share other idea how to detec detail on certain quantizer. (Example shows quant=3)

1)Basic idea (not quite sure, maybe requires modification): high frequency is present on spots where the hi-pass is steeper than original.
So, clips o and q are masks of steepnes and mask is minimum of them.

pixelstring = "1 -1 0 -1 1 0 0 0"
qm=mt_makediff(orig,orig.DCT(1,1,1,1,1,0,0,0),y=3,u=3,v=3)
q=mt_luts(qm, qm, mode = "range", pixels = pixelstring, expr = "y",y=3,u=3,v=3)
o=mt_luts(orig, orig, mode = "range", pixels = pixelstring, expr = "y",y=3,u=3,v=3)
in both of these clips the last line and last column can show edge, so I'll zero all of them using masktools. (interested inside the block only anyway)
#maybe adding one expand here to be sure
mask=mt_lutxy(o, q, expr = "x y > y x ?",y=3,u=3,v=3)
#ringing=mt_lutxy(o, q, expr = "x y > 0 y ?",y=3,u=3,v=3)
make projection of clip mask --> <0;255>

Following steps:
2) shift mask and orig same way
3) make max value of every 8x8 block:
How to do that?
shifted_max=mt_luts(shifted_mask, shifted_mask, mode = "max", pixels = "0 0 .... 8 -8", expr = "y",y=3,u=3,v=3)
shifted_max=shifted_max.PointResize(shifted_max.width/8,shifted_max.height/8)
shifted_max=shifted_max.PointResize(shifted_max.width,shifted_max.height)
Didn't try, but must be pretty slow. Here is done 64 times more operations than necessary. Any faster way?
4) Smooth shifted clip with DCTFilter based on shifted_max
5) shift back

I would appreciate help in steps
1) - qualitywise and speedwise
3) - speedwise

Raziel6969

5th June 2006, 19:08

Hello redfordxx!

I wanted to test your script with a clip from my photo camera, because i got interested with your sample screenshots.
but i got confussed ...

I create .avsi files from:
post #02: Arrays and Loops.avsi, Patterns.avsi, other without name ...
post #16: SmoothDeblock script part one
post #17: SmoothDeblock script part two

... and put in AviSynth plugins folder.
But when I open the script with Virtualdub, I got:
http://img469.imageshack.us/img469/1792/smootherror9uo.png

My script [hdragc.avs]
directshowsource("S4022492.AVI")
HDRAGC()
SmoothDeblock3(density=4)

Questions:
1. What I need to use the script? please be clearer at first post :P
2. Is possible to join all the necessary scripts to become one downloaded .avsi?

Many Regards.

redfordxx

5th June 2006, 19:29

http://img469.imageshack.us/img469/1792/smootherror9uo.png

Strange, it seems to me according to more sources, that once the avsi is autoloaded from plugin dir, the name is somewhat encoded:)

OK, to the topic (I am not home, so quickly)

Questions:
What I need to use the script?
as in post 2
DCTFilter (link should be in your AVS doc)
Average (search Average in thread names only)
please be clearer at first post

I try later;)
Is possible to join all the necessary scripts to become one downloaded
No.
1) updating issue: they can be theoretically updated separately
2) not all are of mine
3) post lenght limit
I recommend joining part one and part two together and thats all.
Functions alone are from my functions.avsi which is not interesting for all, so I extracted it.

Raziel6969

5th June 2006, 20:11

Thanks! The following solved my problem:
DCTFilter (link should be in your AVS doc)
Average (search Average in thread names only)
http://mywebpages.comcast.net/trbarry/DctFilter.zip
http://people.pwf.cam.ac.uk/mg262/posts/blend/Average_21Oct05.dll

from:
http://mywebpages.comcast.net/trbarry/downloads.htm
http://forum.doom9.org/showthread.php?t=100626&highlight=average

Now the script works for me :)

redfordxx

7th June 2006, 10:14

Tobias,

you mentioned your DQI because it "will be released soon" or was it just for information?

...just to know whether to expect something.

708145

7th June 2006, 11:37

you mentioned your DQI because it "will be released soon" or was it just for information?
...just to know whether to expect something.

It's the same soon as with all my fun projects... I never know when I get around to it the next time.

bis besser,
T0B1A5

redfordxx

4th December 2006, 11:16

Hi guys,
I want to continue/rewrite the script with some new ideas. Please, write some opinions on following:

I see two possible changes:

Quantizer decision:
Last time: the only rule was complexity of block measured by destroyed detail on certain quant.
Now I will add another rule: improvement in block smoothing on certain quant.
Result: if it is low complexity block, I can deblock strongly, but if it makes no improvement, why to do it.
Additionally: I am thinking about estimating about relative improvement measuring, which means edge smoothing improvement on shifted DCT compared to total damage on the shifted block.

Adjacent different-quant-blocks dealing:
Algorithm (on shifted block)
1) deblock on highest of all quants
2) replace all blocks with lower quant with the original value
3) check where is block smoothing improvement
4) replace part of blocks with no improvement with the original value
do it again on lower quant if needed

example
original: 0 0 0 0 0 0 0 0|9 9 8 7 3 2 1 0|1 1 1 1 1 1 1 1
quants: 7 3 7
step1q6: 0 0 0 0 1 2 3 5|7 8 7 6 5 4 3 2|2 1 1 1 1 1 1 1
step2q6: 0 0 0 0 1 2 3 5|9 9 8 7 3 2 1 0|2 1 1 1 1 1 1 1
step3q6: yes no
step4q6: 0 0 0 0 1 2 3 5|9 9 8 7 3 2 1 0|1 1 1 1 1 1 1 1
step1q4: 0 0 0 0 1 2 4 7|8 9 8 7 4 3 2 1|1 1 1 1 1 1 1 1
step2q4: 0 0 0 0 1 2 4 7|9 9 8 7 3 2 1 0|1 1 1 1 1 1 1 1
step3q6: yes no
step4q4: 0 0 0 0 1 2 4 7|9 9 8 7 3 2 1 0|1 1 1 1 1 1 1 1
step1q2: 0 0 0 0 1 2 4 7|8 9 8 7 3 2 1 0|0 1 1 1 1 1 1 1
step2q2: no replacement because q7 and q3>current q2

Advice needed:
1. Can someone recommend script or filter to detect discontinuities on block edges?
I use the "-1 3 -3 1" convolution for now, but it is not good enough.
I need something better, something that's not confused by diagonal edges going thru block borders.

2. Still not sure about real edge protection.
* Partially could be of Didée's idea
* partially some logic considering edges presence on adjacent borders
* partially some threshold for magnitude of the edge
* chroma linking...
I don't know

ficofico

4th December 2006, 15:48

Hi redfordxx, I want to try your script but I' haven't understand where's the plugins and How many addon are required?

redfordxx

5th December 2006, 08:30

Hi redfordxx, I want to try your script but I' haven't understand where's the plugins and How many addon are required?

Go through the thread from the beginning, it's all there..

redfordxx

6th December 2006, 15:30

Before I go further, I decided to collect information on existing stuff. Can you help me here (http://forum.doom9.org/showthread.php?t=119082)?

redfordxx

10th December 2006, 00:51

Well, I intend to improve SmoothDeblock and make it dll filter. If I'll be strong enough.
Here is first step. This dll contains two functions
LogDCT()-logs DCT coefficients to file.
FindEdges(Clip, int right, int down, int upright, int downright, int contrast)-finds discontinuities in four directions and makes wighted average of them plus adds contrast.
For example clip.FindEdges(0, 1, 0, 0, 20) discontinuities towards the down neighbour pixel and increases the contrast by 20 (so that we see details)(BTW - this one nicely detects combing)

The approach was:interpolate 6 pixels with 5th degree poly and then compare average of derivations in center pixels with the derivation between center pixels.

redfordxx

10th December 2006, 02:31

The next step woud be estimate the complexity of block from the DCT coefficients.

Reason: as in the script - to know where can I deblock, but now faster and more exact.
I quess it could be some weighted average of the coefficients, maybe squared, I don't know.

Question: It's clear for me, that the more coefficcients are going down right, the more detail. But when the coefficients are in the corners and the center is zeros, does it mean something?

@Tobias: you said you quantized the coefficitent in the SmoothD with SixOfNine and that it is better thant zeroing like DCTFilter. How did you do it? Or better, how should I do it?

redfordxx

18th December 2006, 10:12

Those who were little interrested in what I am doing here might know, that the principle of SmoothDeblock was reducing DCT coefficients on shifted clip and then merging with original weighted so that on block boundary was the shifted clip value, in the block centre was the original and some average was inbetween.

Some attempt to visualize (ax is weighted average of a and x).
original: x x x x x x x x| y y y y y y y y
reduced: a a a a a a a a| b b b b b b b b
weight: 0 96 160 255 255 160 96 0| 0 96 160 255 255 160 96 0
result: a ax ax x x ax ax a| b by by y y by by b

Now, observation (nothing surprising):
Imagine two highly quantized adjacent blocks in frequency domain (with various non-zero values x and y) Then horizontally shifted block looks in frequency domain like the next one (with various non-zero values z):
x x 0 0 0 0 0 0|y y 0 0 0 0 0 0
x x 0 0 0 0 0 0|y y 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0|0 0 0 0 0 0 0 0

z z z z z z z z
z z z z z z z z
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
So, now I see that I need to reduce the coefficients only on right or bottom depending on the shifting direction.

So the question is how strongly to reduce it. My first idea was:
If (i,j) represents (column,row) in the DCT matrix, then
maybe the condition abs(z(i,j))<=max(abs(x(i,j),abs(y(i,j)) can help. There is no sophisticated theory behind but it could work.
Or
abs(z(i,j))<=max(abs(x(i,j),abs(x(i+1,j)...abs(x(7,j),abs(y(i,j),abs(y(i+1,j)...abs(y(7,j))
or
abs(z(i,j))<=max(abs(x(i-1,j),abs(x(i,j)...abs(x(7,j),abs(y(i-1,j),abs(y(i,j)...abs(y(7,j))
For vertical direction just transpose.
Does anyone have other idea. Or some improvement?

redfordxx

18th December 2006, 10:53

I have other improvement idea. Imagine two blocks (spatial):
10 10 10 10 10 10 10 10| 30 35 40 45 50 200 0 200
Apparently, the second block contains very high frequency but only on the half, which is not on the are which will be smoothed together with the left block.
How about extrapolating the left half of the right-hand block like this
10 15 20 25| 30 35 40 45
------------------------------
then test the complexity on the (half) extrapolated area. However, this would involve testing going out of (0,255) range.

But still probably it would be smoothed like
10 10 10 10 10 12 15 20| 22 30 38 45 50 200 0 200
and int would be better
10 10 10 10 10 15 20 25| 30 35 40 45 50 200 0 200
coz definitely the right-hand block was not quantized very much and should not be changed...

redfordxx

22nd December 2006, 12:11

Hi everyone (this was originally intended to Didée, but has full PM-Box so it's here...otoh, anyone can help),

I have still some energy to continue on SmoothDeblock, but now I have problem to solve something. You wrote something about your mathematical background, so if you want to bother a little, pls read further.

My idea is that zeroing hi freq DCT coefficients is not the best way to smooth edges. I have to change the remaining coeffs so that the SSD (comparing original and new texture in spatial domain) is minimal. Which is not the same I believe. Morover, I wanna add weights which will be related to distance of the pixel from the edge and the complexity of block the pixel is member of.

So, what I have:
Definitions
boundaries parameters (defined):
N=8...block size(I know)
C,D...(0,N-1)...where begin zeroed hi frequencies (I know)
C<N or D<N
spatial domain indexes:
i,j...(0,N-1)
frequency domain indexes:
u,v...(0,N-1)
a...(0,C-1)
b...(0,D-1)
pixel intensities, coefficients and weight:
f_ij...original pixel value (I know)
g_ij...new pixel value (I search for)
G_uv...new frequency value (I search for)
w_ij...weight of the SSD (I know)
define abbreviation (so that formulas are shorter):
Z(u,i)=lambda(u)*cos((2*i+1)*(pi*u)/(2*N))
so we have DCT and IDCT formulas like this:
G_uv=2/N*SUM[i=0..N-1 ; j=0..N-1 ; g_ij*Z(u,i)*Z(v,j)]
g_ij=2/N*SUM[u=0..N-1 ; v=0..N-1 ; G_uv*Z(u,i)*Z(v,j)]

Nothing new so far, now set of equations:
(1) if ((u>=C) || (v>=D))
G_uv=0
(2) I want minimize
SUM[i=0..N-1 ; j=0..N-1 ; w_ij*(f_ij-g_ij)^2] = SUM[i=0..N-1 ; j=0..N-1 ; w_ij*(f_ij-2/N*SUM[u=0..N-1 ; v=0..N-1 ; G_uv*Z(u,i)*Z(v,j)])^2]
so derivatives according to G_ab gives C*D<64-7 equations (one equation for each (a,b) combination):
SUM[i=0..N-1 ; j=0..N-1 ;-2/N*Z(a,i)*Z(b,j)*2*w_ij*(f_ij-2/N*SUM[u=0..N-1 ; v=0..N-1 ; G_uv*Z(u,i)*Z(v,j)])]=0
and further from (1):
SUM[i=0..N-1 ; j=0..N-1 ;-2/N*Z(a,i)*Z(b,j)*2*w_ij*(f_ij-2/N*SUM[u=0..C-1 ; v=0..D-1 ; G_uv*Z(u,i)*Z(v,j)])]=0
2/N*SUM[i=0..N-1 ; j=0..N-1 ;w_ij*f_ij*Z(a,i)*Z(b,j)] = 2/N*2/N*SUM[i=0..N-1 ; j=0..N-1 ; SUM[u=0..C-1 ; v=0..D-1 ; w_ij*G_uv*Z(u,i)*Z(v,j)]*Z(a,i)*Z(b,j)]
here is possible to make DCT of weighted original if it helps:
WF_ab = 2/N*2/N*SUM[i=0..N-1 ; j=0..N-1 ; SUM[u=0..C-1 ; v=0..D-1 ; w_ij*G_uv*Z(u,i)*Z(v,j)]*Z(a,i)*Z(b,j)]Here is where I stayed for few hours figuring out that as I don't have 64 equations like this I cannot make I/DCT on right side.
I don't have enough skill to continue and find G_ab. Anyone?

redfordxx

22nd December 2006, 13:00

Moreover, it would be nice to be able to limit solution values interval to
-Limit_ab<G_ab<Limit_ab
but I guess it's too much and not that important.

Nevertheless, the progress in coding is there and according the first test promising at least significant improvement in speed. Won't be lightning fast, but usable.

708145

22nd December 2006, 13:06

redfordxx

22nd December 2006, 13:26

Thanx Tobias,
that was nice;)

so also merry xmas and 'till better;)
R.

dukey

23rd December 2006, 00:15

Personally ..
I dont notice blocking so much in areas of high spatial resolution. In these instances when areas of high spatial resolution are deblocked, it can lead to a really ugly blurry mess.

redfordxx

23rd December 2006, 13:48

Personally ..
I dont notice blocking so much in areas of high spatial resolution. In these instances when areas of high spatial resolution are deblocked, it can lead to a really ugly blurry mess.

I don't see your point:

That should be reaction on something here posted or just a general statement?

dukey

23rd December 2006, 22:08

Well if i was to write a deblocker
I would probably ONLY deblock the areas with very low spatial resolution. Like with xvid most of the picture can look fine but areas which are out of focus like the background of a scene with say a colour graduation. This will often lead to blocking and mach banding type artifects. This will make your eye see things which shouldn't really be there. You won't notice these mach banding type effects so much in areas of high spatial resolution anyway .. and deblocking areas with high spatial resolution can just murder the detail.

You could probably pick out areas of low spatial resolution simply by seeing how close in colour the neighbouring blocks are.

I've never seen a deblocker do what I have mentioned above, but I think it might be worth it. I think the 'problem' a lot of deblockers is not that they don't work, but they often deblock parts of the image (killing the details) which often don't need it. Or you dont see many artifacts in these places.

just my thoughts

redfordxx

26th December 2006, 17:17

mach banding type artifects.What do you mean by that?

You could probably pick out areas of low spatial resolution simply by seeing how close in colour the neighbouring blocks are.IMO that's not that easy:
you can have highly quantized (blocky) blocks with big block-to-block color difference. (I think it happens often on keyframes with not enough bitrate)
you can have very similar colors with high detail (yellow sand e.g.)

I think the 'problem' a lot of deblockers is not that they don't work, but they often deblock parts of the image (killing the details) which often don't need it.I mostly agree with you but I would say that it is not because they deblock 'all'. They often try to decide where to deblock and where not, but they cannot distinguish it well.

What I want here, is to be able decide the level of detail according to quantization coefficients.

Or you dont see many artifacts in these places.There is another point in deblocking than preventing you seeing the artifacts: making easier source for next compression. Because recompression (which is what are all the Avisynth deblockers meant for) often involves cropping or resizing and that brings the blocking edges in the middle of the new blocks which will eat lot of bitrate which will cause additional blocking somewhere else.

dukey

29th December 2006, 11:05

your eyes naturally enhance edges
mach banding is this sort of. Happens in areas of different brightnesses.
Imagine a colour graduation and reducing the colour depth to 16bit. It'll look horrid and your eyes will enhance the edges where the brightness changes making it look even worse than it actually is !

The point im trying to make it. In areas of high spatial resolution (detail) if there is blocking, you probably wont notice it so much as it is surrounded by a lot of other detail. But in areas where there is very low detail, you will create obvious artifacts which will stand out.

Take a look at this pic .. just something i snapped from an xvid encode. Was encoded at pretty low bitrate.

http://img153.imageshack.us/img153/6649/deletens3.png

Look at the sky .. your eyes are enhancing the edges of the blocks, even though the colours maybe pretty damn close together.. By deblocking only those areas of with very low detail, you might be able to get good results without destroying the picture and making it look like a blurry mess.

I am no expert in deblocking, these are just my observations as a n00b :)

redfordxx

25th November 2007, 23:21

seems to have found time again to work on it:

I have been thinking about the deblocking strength algo... it decides on every place in the frame the strength of deblocking. But, if the frame is encoded on one quantizer it is not necessary. The whole frame should be deblocked with one strength. It can be sognificant speedup.

However, not all codecs use constant quant all over the frame.
My current feeling is that MPEG4 uses constant quant per frame and MPEG1 and MPEG2 do not.
1. Can anyone confirm or correct this?
2. Is it possible to quess somehow the logic of MPEG2 or MPEG1 encoder to to better decide deblocking strength?
I mean:
- a block having high frequencies means low quantization --- thats easy
- a block having high frequencies means low detail in original before compression or high quantization during the compression? How to decide this? Could in the decision help knowing the level of detail in neighbouring blocks, average detail of the frame, min detail of the frame, max detail, color of the block...whatever?

I had similar question here (http://forum.doom9.org/showthread.php?t=131929)...

Tnx 4 ur ideas
R*

MfA

26th November 2007, 01:39

High frequencies can mean an edge, in which case you want a low quantizer ... or it can mean noise or texture, in which case you want a high quantizer. Also expecting the encoders to make sane choices all the time is probably way too optimistic.

I still think your efforts would be better spend in a place where the data you want is actually available ... try taking a look at DMPGdec, writing a drop in replacement for it's post processing is relatively easy, and the postprocess function already gets an array with the per block quantizer values.

redfordxx

26th November 2007, 21:13

High frequencies can mean an edge, in which case you want a low quantizer ... or it can mean noise or texture, in which case you want a high quantizer
Well, I don't know what do you mean by texture here, but speaking about noise...I am not interested in denoising at this moment, so if I am not mistaken: presence of noise=>high freq=>was low quant=>no blocking...

writing a drop in replacement for it's post processing is relatively easy, and the postprocess function already gets an array with the per block quantizer values.
Confused...my english is probably not good enough;-)

MfA

26th November 2007, 21:17

Take DGMPGdec source code, replace postprocess with your own deblocking code, compile, run.

redfordxx

27th November 2007, 00:30

Aaa, that's what writing a drop means... Well, how about only adding an option to the MPEG2Source to produce the mask of quantizers? I could do then whatever i want with that later...

Then I really like to have confirmed from someone, whether there are other codecs with variable quant within one frame than those which can be processed by DGMPGdec...

foxyshadis

27th November 2007, 03:16

Sure, practically any codec can use adaptive quantization, and they can be stored in most containers. Not all implementations do, but it's very common to find MPEG (1-4) and VC-1 with quants varying per-block.

You have to take into account the quant matrix, if you really want to figure out what quant was there before, because a 'high detail' matrix can cause a lot of blocking while also keeping a lot of noise/texture. (Texture means a pattern, though it doesn't have to be regular; here it means all of the useful non-edge information.) High quants with these matrices cause a lot of ringing/mosquito noise, instead of just removing high frequencies.

redfordxx

27th November 2007, 10:57

Sure, practically any codec can use adaptive quantization, and they can be stored in most containers. Not all implementations do, but it's very common to find MPEG (1-4) and VC-1 with quants varying per-block.
interesting...I tested few codecs with show quants option in FFDShow and X264, XVID, MP42 had constant and MPG1, MPG2 had varying...

You have to take into account the quant matrix, if you really want to figure out what quant was there before, because a 'high detail' matrix can cause a lot of blocking while also keeping a lot of noise/texture. (Texture means a pattern, though it doesn't have to be regular; here it means all of the useful non-edge information.) High quants with these matrices cause a lot of ringing/mosquito noise, instead of just removing high frequencies.OK... the point is, when I know quant and matrix for block, i can calculate error or compression information loss, right?
error_uv=quant*m_uv/2
error_uv=max error for frequency uv
m_uv=value in quant matrix
...at least for non AVC keyframes, right?

foxyshadis

28th November 2007, 00:26

interesting...I tested few codecs with show quants option in FFDShow and X264, XVID, MP42 had constant and MPG1, MPG2 had varying...

If you never use the capabilities, then AQ won't be present, obviously. You can make your own by enabling that option in xvid, x264 (patched), mainconcept h.264, Nero, or various ffdshow codecs (in the 'Masking' option page).

redfordxx

29th November 2007, 10:30

Hi,
after reading some MPEG-2 docs and thinking over I got following (pls comment whether I am mistaken if you like):
- maybe I know what is pattern already. extreme example: when all freqs are veeeery strongly quantized except the last one, then on decoded frame the last frequency can dominate and we see it as 2D wave…thats the pattern…? This example does not happen because the coefs are increasing to higher freqs…but is that the principle.
- the amount of detail in decoded block does not say anything about the possible error=>blocking magnitude
- the amount of detail in decoded block does say about the possible damage when zeroing hi-freq coefs with DCTFilter on shifted video
- if I learn the quantizer from DGDecode and read the Qmatrix, I can calculate various values and averages of max or expected error…this could help for deblocking adaptability
- this calculation is same way possible for I frames or P frames (except the [0,0] coef?), I don't know how about B frames
- surprise: it is not that a frame DCT is either frame based or field based. My feeling is it can differ every macroblock. Some blocks are encoded progressive, some interlaced within one frame…
R*

redfordxx

1st December 2007, 01:09

Hi, again begging for opinions:
I played with excel and simulated the codec process using MPEG quant matrix I obtained from DGDecode and random 8x8 sample blocks with following results (maybe I am discovering discovered thing but thats the risk:):
I defined TME(Q)= (theoretical maximal error for quantizer Q) = (value of pixel[0,0] if all values of freq coefs before inverse quantization equal 0.5)
example:
TME(1)=7
TME(2)=12
TME(3)=19
TME(4)=24
TME(5)=30

Maybe my calculation is not 100% MPEG compliant, but roughly so...
Now all calculated values can be formed as multiplication of this TME for desired quantizer.
Result looks like:
EA(Vert)=expected absolut value of average of absolute values of pixel errors on vertical edges
E(Vert)=expected absolut value of average of pixel errors on vertical edges
MA(Vert)=maximum absolut value of average of absolute values of pixel errors on vertical edges
M(Vert)=maximum absolut value of average of pixel errors on vertical edges
....and
E(Vert)=1.6%
E(Hor)=1.7%
M(Vert)=6.2%
M(Hor)=6.4%

EA(Vert)=7%
EA(Hor)=6.7%
MA(Vert)=14.1%
MA(Hor)=13.3%

M(Max8x8)=47%
E(Max8x8)=28%
and so on

so then, when processing the video:...macroblock with quant 10 comes, I know:
TME(10)=60
E(Vert)=1.6%*60=0.96
E(Hor)=1.7%*60=1.02
E(Hor)=6.4%*60=3.84
..and even per pixel if needed
E(pixel[0,0])=6.9%*60=4.14
M(pixel[0,0])=24.61%*60=14.766
E(pixel[1,0])=8.65%*60=5.19
M(pixel[1,0])=29,28%*60=17.568

The problem of the theory can be here

E(Hor)*TME(40)=1.7%*242=4.114
M(Hor)*TME(40)=6.4%*242=15.488
M(Max8x8)*TME(40)=47%*242=113.74

...estimated maximal possible error is 113. Is it really possible in real world or in such cases encoder chooses lower quant?

Conclusion: I think, this can be together with with edge detection solid base for adaptive deblocking in my filter. However
1) I modelled the samples simply =INT(RAND()*256), which is not reality...in reality the samples would be correlated and the error will be smaler. How to model more real-like situation, any ideas?
2) I hope there are no errors like the 113 example or higher. I guess the encoder chooses lower quant but how to estimate this. Is there any SNR or block complexity/variance rule which can help?

Thanks for ideas.

R*

redfordxx

2nd December 2007, 02:00

Step one finished: I have the quantizer mask from MPEG file from here (http://forum.doom9.org/showthread.php?t=132326);-)

Now...can anyone guess the distribution of the error values in non-intra block? Or at least some average...or something. What I think is, that the values are small compared to intra values but can be negative... but any better estimation?

redfordxx

3rd December 2007, 14:35

Can you tell me opinion how to make weighted smoothing correctly?

If we have weight clip w and values clip x then (symbolically):

s="1 2 3 2 1" #weights of moving average
y=w*x
z=y.convolution(s)
ww=w.convolution(s)
output=z/ww
this is clear.

But how about smoothing using DCTFilter?

y=w*x
z=y.DCTFilter(1,1,1,1,0,0,0,0) #for example
#now which formula for ww is correct?
ww=w
ww=w.DCTFilter(1,1,1,1,0,0,0,0)
ww=w.DCTFilter(1,0,0,0,0,0,0,0) #probably not
output=z/ww

Thank you.

redfordxx

12th December 2007, 17:52

@Tobias (or whoever else can answer)

Once you mentioned, that for smoothing in my filter, quantization with the original matrix which was used to encode would be better than using DCTFilter.
I am thinking about implementation. I plan to read DGDecode quants, and DGIndex quantmatrixlog, so it is possible. But can you please explain the reasons, so I know whether it is worth the effort?

If I finish, expect improved MMX SmoothDeblock, so no more speed issues… so I wanna do it well.

708145

13th December 2007, 16:18

Hi redfordxx,

good work. Keep up the speed :D

Once you mentioned, that for smoothing in my filter, quantization with the original matrix which was used to encode would be better than using DCTFilter.
I am thinking about implementation. I plan to read DGDecode quants, and DGIndex quantmatrixlog, so it is possible. But can you please explain the reasons, so I know whether it is worth the effort?

If you the original matrix and original quant values you keep the same detail that was retained during encoding. If you use a "smoother" matrix it would blur too much.

Could you post performance figures for your MMX code once you get them? It would be great to use that for playback on HD resolutions :)

bis besser,
708145

redfordxx

13th December 2007, 17:00

If you the original matrix and original quant values you keep the same detail that was retained during encoding.
Even if used shifted?

There will be the option to smooth with larger matrices...I think 12x12 or 16x16 might be interesting.
How to apply the rule you mentioned on that? What should look a matrix bigger than 8x8 like to smooth similarily like a the known 8x8 matrix?
Seems to me that the down right corner could look similar...but I dont know

Could you post performance figures for your MMX code once you get them?Well there is still a way to go...
It would be great to use that for playback on HD resolutions :)The thing is, I intend to make this filter two pass:
* first pass, which would be very fast, will be done on a clip with showQ=true and saved (look at the link few posts above)
* second, the real deblocking will happen

Soulhunter

13th December 2007, 17:51

The thing is, I intend to make this filter two pass:
* first pass, which would be very fast, will be done on a clip with showQ=true and saved (look at the link few posts above)
* second, the real deblocking will happen

Well, for realtime processing the 2 steps could be interleaved, no?

Bwt, good to see some movement in here... :]

Thx n' Bye

redfordxx

14th December 2007, 19:35

If you the original matrix and original quant values you keep the same detail that was retained during encoding. If you use a "smoother" matrix it would blur too much.I still don't see it:
Whatever frequencieds the matrix and quantizer quantize, it always results in high frequencies in shifted block. And these high frequencies should be partially/fully removed. Or it is different?

708145

15th December 2007, 22:10

Even if used shifted?
There will be the option to smooth with larger matrices...I think 12x12 or 16x16 might be interesting.
How to apply the rule you mentioned on that? What should look a matrix bigger than 8x8 like to smooth similarily like a the known 8x8 matrix?
Seems to me that the down right corner could look similar...but I dont know

Well this is a tough question as I have not tried that. But guessing from the fact that larger blocks tend to result in more ringing, but the ringing averages out with the shifting approach I have truely no clue :p

I still don't see it:
Whatever frequencieds the matrix and quantizer quantize, it always results in high frequencies in shifted block. And these high frequencies should be partially/fully removed. Or it is different?

Those high frequencies that define sharp edges have to be kept. And those frequences will be treated in the same way as by the original matrix, so the look and feel stays the same.

bis besser,
708145

foxyshadis

15th December 2007, 22:14

If you remove all or most high frequencies, you may as well just use Deblock() or fft3dfilter(). In fact I can't really find any way around the inevitable detail-or-noise aspect of ringing, except in special conditions like anime.

MfA

15th December 2007, 22:42

Ringing is a harder problem, but deblocking doesn't need to do damage to detail (although the approach chosen here isn't good in that respect).

redfordxx

16th December 2007, 05:52

Those high frequencies that define sharp edges have to be kept. And those frequences will be treated in the same way as by the original matrix, so the look and feel stays the same.I think we are talking about different thingsOriginal: Highly quantized - blocking:
o |
o | |
o | o o o o |
o | o o o o|
o | |
o | |
o | |
o| |
|o |
| o |
| o |
| o |
| o |o o o o
| o | o o o o
| o |
| o |
high frquencies here ^
Removed hi freq on shifted block:
|
|
|
o o o o |
o |
o o |
o|
|
|
|o
| o o
| o
| o o o o
|
|
|
| shifted block |

MfA

16th December 2007, 10:03

High frequencies aren't "here" ... at least not with DCT, which has a very coarse localization in space (undecimated wavelets are better in that respect). The shifted block will have energy in high frequencies if there is blocking, also if there is texture, also if there are high contrast edges other than from blocking.

If you can't distinguish, you're gonna kill it all ... even if you can distinguish, you can't selectively kill it with this approach.

PS. blocking isn't really something restricted to high frequencies with DCT, because it can not capture a smooth gradient sparsely ... the energy gets spread out over the entire frequency range. Killing the high frequency content will reduce the blocking, it turns the hard edges into softer edges, but it won't give you a smooth gradient.

redfordxx

16th December 2007, 12:20

High frequencies aren't "here" ... at least not with DCT, which has a very coarse localization in space (undecimated wavelets are better in that respect).Hmh...don't understand...

The shifted block will have energy in high frequencies if there is blocking, also if there is texture, also if there are high contrast edges other than from blocking.

If you can't distinguish, you're gonna kill it all ... even if you can distinguish, you can't selectively kill it with this approach.

PS. blocking isn't really something restricted to high frequencies with DCT, because it can not capture a smooth gradient sparsely ... the energy gets spread out over the entire frequency range.Of, course...this was only simplified example for the discussion about the Tobias'es statement "Those high frequencies that define sharp edges have to be kept"
Killing the high frequency content will reduce the blocking, it turns the hard edges into softer edges, but it won't give you a smooth gradient.Well, nothing is perfect, unless some great ideas appear here;-)
Ringing is a harder problem, but deblocking doesn't need to do damage to detailHm, but all deblockers I know so far either do not deblock enough or damage detail:-(

Dark Shikari

16th December 2007, 12:32

redfordxx

16th December 2007, 13:25

A thought--why not remove all high frequencies from the image, then deblock the low frequencies, then re-add all the high frequencies?

This of course would require that the blocks line up with your frequency-removal.This is also one option I have in mind. However, removing high frequencies from DCT would not be OK, at first sight.
It would be necessary to ensure that the borders stay where they are (solving 28 equations with 28 variables --- maybe less when decided not all pixels should be exactly same) plus maybe some more equations to ensure the smoothed block is similar to the original using SSD. I plan to include it as one of the features of my new DCTFilter...if I find some good algo.

The good thing is, this could be separate plug-in and it could be used as preprocessor for any existing deblocker. Didee does something similar in Deblock_QED

MfA

16th December 2007, 13:55

Hmh...don't understand...
Frequencies in the DCT transform don't represent localized features.
Hm, but all deblockers I know so far either do not deblock enough or damage detail:-(
Most deblockers apart from SPP (but including this one) are effectively selective blur filters. If you want to improve on what's out there then selective blurring is not the way.

You always run the risk of confusing blocking with image edges at the block edge, but there is no reason detail inside blocks should be damaged by deblocking.

SPP would be a good starting point, one quick and easy way to try to improve on it would be to get it to respect quantization bounds (although doing it at the end probably wouldn't produce good results).

Dark Shakiri's, that is only going to help on blocks which are relatively smooth to start with ... the high frequency quantization artifacts also cause blocking.

redfordxx

17th December 2007, 20:50

Frequencies in the DCT transform don't represent localized features.Meaning...presence of hi freq does not imply amount of detail?

Most deblockers apart from SPP (but including this one) are effectively selective blur filters. If you want to improve on what's out there then selective blurring is not the way.Well, you seem knowing more of the stuff than me...so what would be the way?

SPP would be a good starting point.I tried to google it little but didn't find the algo. The one thing I found is that it is similar to SmoothD...well SmoothD having similar principles like Smoothdeblock, does what I understand under blurring. And I don't think it is selective...

Damn, I posted this already yesterday but didn't work...so this is fast repro, hope it makes sense... I am in Brussels now, wow guys the keyboards here are really funny;-)

MfA

17th December 2007, 23:36

Meaning...presence of hi freq does not imply amount of detail?
Well, no. It can be texture, it can be original edges, it can be blocking (assuming a shifted transform).

But this is making too much of a deal about a very simple remark ... you said there were high frequencies "here" at the blocking boundary and I was simply saying you couldn't know they came from there (or anywhere specific in the block).
I tried to google it little but didn't find the algo. The one thing I found is that it is similar to SmoothD...well SmoothD having similar principles like Smoothdeblock, does what I understand under blurring. And I don't think it is selective...I was talking about Mplayer's SPP (http://svn.mplayerhq.hu/mplayer/trunk/libmpcodecs/vf_fspp.c?revision=25249&view=markup).

MSE/PSNR wise it can still be improved (and made even slower) by having it respect quantization boundaries.

redfordxx

18th December 2007, 08:43

Well, no. It can be texture, it can be original edges, it can be blocking (assuming a shifted transform).So, it can be said that if the shifted block has higher freqs than original block then it is blocking...maybe?

708145

18th December 2007, 13:30

So, it can be said that if the shifted block has higher freqs than original block then it is blocking...maybe?

Only with some probability.
Because it could still be that a real steep gradient is located at the block border. Then you have to make sure this doesn't get deblocked.

redfordxx

18th December 2007, 21:28

I was talking about Mplayer's SPP (http://svn.mplayerhq.hu/mplayer/trunk/libmpcodecs/vf_fspp.c?revision=25249&view=markup).Well, I don't think I am strong enough to read it right now...is maybe some algo/paper describing it?

redfordxx

18th December 2007, 23:10

I see that you have many ideas, but writing me only what is not possible...so let's get some work done, guys...

I sumarize what info we have available:
1 - I can read quantizer values from DGDecode for each macroblock
2 - I'd like to implement reading quant matrix from the file which is produced by DGIndex (if someone recommends me where can I use some opensource code, I'd appreciate)
3 - If I know 1 and 2, I can estimate for each macroblock whether it is I-block progressive, I-block interlaced or something else (just by trying to apply the matrix and quantizer I have
4 - I believe I can estimate the mean and max error of each pixel, whole block or block border as a function of quantizer and variance of the pixels for given matrix (but this is still to be explored)
5 - I can probably measure the magnitude of edges on block borders. I am still not sure which method is most suitable...I mean is not confused by edges going across or diagonally across the borders (what do you think?)
6 - I can calculate something like second derivation inside the block to detect the detail
7 - In case of blocking, there is probably some relationship among edges on various borders of the same block or edges connected to each other. Let's use it... although now I don't know how;-)

There more types of solutions I thought of:
- first maybe as Dark Shikari mentioned, remove some detail
then
1) perform smoothing by weighted quantization (weights=expected errors, strenght=derived from the quants form the dgdecode showQ information, quantization can be substituted by just removing high frequencies, this can maybe be limited by max expected error)
2) find completely new freq coeficients by using SSD method while having some high frequencies zeroed;-)...well, system of linear equations...lot of them

[OK, third time writing this message and trying to post it (I am still in Brussels' hotel and the internet works weird here)]

Leak

21st December 2007, 12:22

Well, I don't think I am strong enough to read it right now...is maybe some algo/paper describing it?
A bit of Google-fu after noticing the URL in the comment at the top of the code isn't valid anymore... (http://citeseer.ist.psu.edu/118727.html) (click the "PDF" link at the top right of the page)

np: Yello - She's Got A Gun (Live At The Palladium NY Sep. 1985) (Claro Que Si)s

MfA

21st December 2007, 15:11

1) perform smoothing by weighted quantization
The high frequency quantization = smoothing meme is rather dangerous ... it's only true when the coefficient is actually zero'd, otherwise it simply introduces quantization noise. When you are not coding you usually only want the first effect and not the latter, so you are better off just thresholding (ie. making the coefficient zero if it's below a threshold, leaving it alone otherwise). SPP being the exception ...

Anyway, I stumbled across a rather original use of 16x16 DCTs for deblocking. It provides a way to determine which high frequencies are probably original image content and which to remove for deblocking.

http://stevehoelzer.dyndns.org/projects/capidd/

redfordxx

22nd December 2007, 20:27

@MfA:
I looked only shortly on the link..will study up. Isn't is something similar to this (http://forum.doom9.org/showpost.php?p=913708&postcount=10)?
Again...with the problems on non I-frames?
Well anyway..will read it...

MfA

22nd December 2007, 21:27

No not really similar, in that paper (http://www.iti.gr/db.php/en/publications/details/445.html) they minimized a blockiness measure for each DCT coefficient separately to get weights for predicting the "least blocky coefficient" which they then clipped to the quantization bounds (without knowing the quantization bounds it just doesn't work).

Capidd uses a shifted DCT across the blocking boundaries (just like you) but it has a novel way to try to determine which of the high frequencies in the block are likely causing the blocking. It will probably work decently even without knowing exactly how the frame was quantized. It's pretty close to what you wanted to do.

redfordxx

24th December 2007, 00:46

Hmm, after reading it...I like it. I will play with it a little;-)
What I see already now:
-it will desire perfectly deinterlaced source. Or some other trick can help. But as combing changes per each macroblock, the postprocessing cannot be easily applied for each whole field separately. Anyway...I was thinking about decomb algo with similar type of prediction some time ago.
-on Figure 8(b), when there is strong blocking, the rest of blocking (although smoothed) is visible. I experienced similar problem in SmoothDeblock...maybe the weighting/averaging could be improved...
-maybe also the prediction could be improved using the quantizer and QM info... (not just 0/1 weighting)...but this is just thinking too much in advance

Max73

12th August 2010, 19:43

Hello redfordxx,

I just have been playing around with your script ("SmoothDeblock3.avsi"), and I find it AWESOME!, no comparison with MSU smart deblocker, or even with bilateral filtering,
neither of the 2 manage to remove strong blocking artifacts near true edges, which still appear jagged.
With your approach instead such artifacts totally disappear,
while most of the true detail is preserved!

I have also played with temporalDegrain, which can also attenuate artifacts strongly, but that one is slower and kills detail to a large extent. Even for heavily compressed material, where you expect little detail to be irrelevant, i still prefer the results with SmoothDeblock3.

About the parameters (which go to the function BorderMask), this are the most important conclusions i could draw:

#bth2 : Suggest 96: the higher the more detail preserved; 100) seems to preserve quite some detail yet removes blocks
#bshare : no effect
#eth1,eth2 : no effect
#eshare : keep 0 or get no deblocking at all

Just AMAZING! Great work :-)!:thanks:

koopa

16th August 2010, 04:43

Really? With this enthusiastic resurection, I tried Smoothdeblock3() only to get a:
"there is no function named "matrix" on line 164 and 349"
error.

Perhaps I have unsuccessfully made a Smoothdeblock.avsi by splicing together the quotes in posts 16/17?

Didée

16th August 2010, 07:52

Have a closer look at post#2. ;)

koopa

16th August 2010, 14:46

Have a closer look at post#2. ;)

It's alright, I still have my dignity *smacks head*

StainlessS

21st January 2012, 19:20

Had to dis-inter this to say 'THANKYOU REDFORDXX', excellent work here.

redfordxx

22nd January 2012, 18:18

If you tried it recently (I didn't) can you say anything about the speed? I cannot remember now, but I guess it is not very fast.
Because there might be some speedup possibilities. It is just issue of my free time to do it. For example:
current RAverage is much faster than the former one
patterns.avsi maybe can be probably optimized

StainlessS

23rd January 2012, 04:57

I did not time it, but think it was in the region of 3-4FPS for PAL DVD sized clip.
(3Ghz single core, VDubMod, In/Out lossless YV12 UT Video Codec on same I/O IDE drive).
Was not really too concerned with the timings, I did it on a spare machine
and left it running all night, and auto shut down when complete. I guessed that
RAverage would be faster, but did not investigate that.
One thing of note though, 3 of the links in 1st 1 or 2 posts are dead and the
4th one (I think) points to a page where the link there is dead. I had all the plugs
already anyway though so was not a problem for me.
Thanks again for a nice set of scripts, probably a little too sedate for everyday
usage though. :)

EDIT: Quote from above, "auto shut down when complete", see here:-
http://forum.doom9.org/showthread.php?p=1520680#post1520680

EDIT: Also got a CaviStream (or whatever its called) access violation when I tried
it with YUY2, perhaps there should be a colorspace check.

Jenyok

23rd January 2012, 08:28

redfordxx (красный форд хх)

Could We see full script or dll (SmoothDeBlock and so on) and needed for it all *.avsi, *.dll, script and other files in one post here ?
i.e. All in One...

Thanks a lot.

librarian

23rd January 2012, 17:53

Here (http://www.2shared.com/file/u-RIoEqJ/SmoothDeblock.html) it is.

Bloax

24th January 2012, 10:25

Yesterday I couldn't get the file, so if that happens - here's another link:
linketylinkety (http://www.mediafire.com/?6x297gf7xivv7uz)

redfordxx

24th January 2012, 16:02

Thank you guys for posting it. Looks like someone is still using it, it's nice;-) I noticed that you joined all scripts into one which I don't consider very wise but whatever you think...
I tried SmoothDeblock3 followed by 3-frame mocomped dfttest on 1080p but it is DAMN slow in this combination... but the result looks really nice;-)

@Jenyok: no ja net avtomobil! I kak "xx" v azbuke???

Jenyok

24th January 2012, 19:45

redfordxx

Красный - red
форд - ford
xx - xx
And all togather...
Красный форд xx...