MaxDeblock - brutal, adaptive, high detail, experimental, slow, unfinished challenge

[redfordxx]

This deblocker attempts and wannabe:
brutal = it can deblock almost everything
adaptive = highly quantized areas will be deblocked & blurred more
high detail = all the real detail should be retained
experimental = there are many ways to use and implement this deblocker's basic idea, it's needed to find the best one
slow = it is now slower than it is necessary (there are ways to speed-up 1/some speed/quality trade off 2/optimization after it is really finished and tuned
not finished = two things: 1/ improve the algorithm coz this is very draft version 2/ lot of tuning necessary
challenge = if you find the idea interesting, I'll be glad for help, because there are things I am only trying to use correctly - example:
1/ DCTFilter - I do not know exactly how it works so I am trying
2/ MPEG codec algorithms - some more knowledge could maybe help somewhere
3/ Tuning on different sources - my hypothesis is that for one person's taste there should be one setting working on all sources.

Principle:
1/ Brutally deblock and smooth everything
2/ Guess quantization of source and produce mask(s) of detail amount
3/ Adaptively restore detail (in three or four stages - in example are four)
2a/ Restore low frequencies everywhere
2b/ Restore medium frequencies in medium quantized areas
2b/ Restore high frequencies in little quantized areas
2b/ Restore original in "nonquantized" areas

Following remarks are mainly for me but if you can help with ideas, welcome:
To do / to solve:
- retain real edges on block borders in highly quantized areas
- high-quantized but still high detail blocks (I guess this can happen due to motion compensation) can be misjudged for low-quantized blocks
- smooth edges correctlywhere is high detail block next to highly quantized block
- do hi-quantized blocks shift average values of high detail neighbours? Shouldn't
- sci-fi?: deringing incorporation, blocks on diagonal edges treatment
- tuning, tuning, tuning
- luma and chroma need probably diferent thresholds...

Questions:
- is DctFilterD better for this than DctFilter?
- is BlockBuster("show") or other filter more suitable for detail detection?

*******************************************
This is first version script:
The parameters are quite fast guessed, probably not enough detail retention. Should be changed.
MaxDeblock.avsi
###############################################
#
# MaxDeblock 0.00000000000000000001 by Redford
#
# imported Patterns.avsi
# plugged mt_masktools.dll
#
###############################################

function MaxDeblock1(clip "orig")
{
b0=orig.BlindPP(quant=31,cpu=4,moderate_h=1, moderate_v=1).Deblock().mt_convolution("1 1 1 1 1","1 1 1 1 1",y=3,u=3,v=3)

d0=mt_makediff(orig,b0,y=3,u=3,v=3)
d0b=d0#.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)
c0=d0b.mt_convolution("1 1 1 1 1 1 1","1 1 1 1 1 1 1",y=3,u=3,v=3)
b1=mt_adddiff(b0,c0,y=3,u=3,v=3)

d2=mt_makediff(orig,b1,y=3,u=3,v=3)
d2b=d2#.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)
c2=d2b.mt_convolution("1 1 1 1 1","1 1 1 1 1",y=3,u=3,v=3)
b3a=mt_adddiff(b1,c2,y=3,u=3,v=3)
q2=orig.QuantMask(7,0,10,0)#.mt_convolution("1 1 1 1 1","1 1 1 1 1",y=3,u=3,v=3).mt_lut("x 4 *",y=3,u=3,v=3)
b3=mt_merge(b1,b3a,q2,y=3,u=3,v=3)

d3=mt_makediff(orig,b3,y=3,u=3,v=3)
d3b=d3.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)
c3=d3b.mt_convolution("1 2 1","1 2 1",y=3,u=3,v=3)
b4a=mt_adddiff(b3,c3,y=3,u=3,v=3)
q3=orig.QuantMask(6,0,10,0).mt_convolution("1 1 1 1 1","1 1 1 1 1",y=3,u=3,v=3).mt_lut("x 4 *",y=3,u=3,v=3)
b4=mt_merge(b3,b4a,q3,y=3,u=3,v=3)

d4=mt_makediff(orig,b4,y=3,u=3,v=3)
d4b=d4.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)
c4=d4b
b5a=mt_adddiff(b4,c4,y=3,u=3,v=3)
q4=orig.QuantMask(4,0,10,0).mt_convolution("1 1 1 1 1","1 1 1 1 1",y=3,u=3,v=3).mt_lut("x 4 *",y=3,u=3,v=3)
b5=mt_merge(b4,b5a,q4,y=3,u=3,v=3)

return(b5)
#return(d0).mt_lut("x 128 - 10 * 128 +",y=3,u=3,v=3)
}
And this (http://forum.doom9.org/showthread.php?p=825073#post825073) is necessary.

[foxyshadis]

At this point I'm fairly sure that combining EEDI2 and YourFavoriteDeblocker will be both faster and better looking. Maybe this is part of a ploy to get me to release BlockTerminator :p but the basic premise is (lot of stuff chopped out):

DeBlock_QED(quant1=30,quant2=51)
Spline16Resize(m(16,width*1.5),m(16,height*1.5))
LimitedSharpenFaster()
EEDI2().TurnRight().EEDI2().TurnLeft()
FFT3DFilter(sigma=1,bt=3,sharpen=0.7)
Spline16Resize(dest_x,dest_y)
AddGrain(noise,0,0)

Now, perhaps your detail-preservation methods can help more than tossing in some fake grain, but if you have blocking that bad, probably isn't worth salvaging much. EEDI has the nifty quality of reconnecting and reconstructing edges, amazing for animation and still cool for natural video, which is where most other deblockers run up to the limit of their ability. The next rev of EEDI2 should be even better suited for it.

Deblock_QED needs to be swapped out for something stronger right now though, redeblock and msu smartdeblock are what I'm considering now.

[redfordxx]

but if you have blocking that bad
most of clip is typically quite OK, but there are high motion parts or darker parts, which need more deblocking
EEDI has the nifty quality of reconnecting and reconstructing edges, amazing for animation and still cool for natural video, which is where most other deblockers run up to the limit of their ability. The next rev of EEDI2 should be even better suited for it.Well, never heard of EEDI...my faultDeblock_QED needs to be swapped out for something stronger right now though, redeblock and msu smartdeblock are what I'm considering now.
I tried other deblockers but all had some problem:
QED - weak
BlindPP - detail destruction
Deblock - 2 lines thick pattern I mentioned in some thread here
MSU Deblocker - Shifts chroma even when chroma is off and I think was not strong enough (didn't checked latest version) I also started a thread about it...

Be careful: ReDeblock is not tested enough.



Deblock_QED... maybe replacing DctFilter with LinearBlock would help...

[danpos]

function MaxDeblock1(clip "orig")

should be

function MaxDeblock1(clip orig)

d0b=d0#.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)

is the same than

d0b=d0

???

Same question for:

d2b=d2#.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)

Cheers,

[redfordxx]

should befunction MaxDeblock1(clip orig)Dunnno, it just works...d0b=d0#.BlindPP(quant=30,cpu=4,moderate_h=30, moderate_v=60)As I wrote, it's experimental version, so you can uncomment it and see whether it is better or only slower.