Deblock_qed Error

[mrkazador]

Using Deblock_qed for my mpeg2 source(HD) i get an error...

Crop: You cannot use crop to enlarge or shift a clip.

I searched for an answer and came up with this thread.
THREAD (http://forum.doom9.org/showthread.php?t=107413&highlight=deblock_qed)

I tried to figure out what exactly to do but I just cannot figure it out. I know its something very simple yet im trying to hard. If someone would be nice enough to correct the avs so I could use it for an High Definition Mpeg2 that would be great. Thank you
function Deblock_QED ( clip clp, int "quant1", int "quant2",
\ int "aOff1", int "bOff1", int "aOff2", int "bOff2", int "uv" )
{
quant1 = default( quant1, 20 )
quant2 = default( quant2, 40 )

aOff1 = default( aOff1, quant1/2 ) # I've no clue if these are clever values or not!
bOff1 = default( bOff1, quant1/4 ) # So:
aOff2 = default( aOff2, quant1/4 ) # Also try all these 4 values @ 0 (zero),
bOff2 = default( bOff2, quant1/8 ) # and quant1=30, quant2=40~45 instead.
uv = default( uv, 1 ) # u=3 -> use proposed method for chroma deblocking
# u=2 -> no chroma deblocking at all (fastest method)
ox = clp.width() # u=1|-1 -> directly use chroma debl. from the normal|strong deblock()
oy = clp.height()

# With avisynth scripting, there is no information available about the position of the currently
# processed pixel ... there simply is no such thing like an "actual" processed pixel.
# So first I've to build up a grid covering the transitions between all 8x8 blocks,
# and make some LUTmania with it later. Yes, this is cumbersome.

block = blankclip(clp,width=6*4,height=6*4,color=$000000).addborders(4,4,4,4,color=$FFFFFF)
block = stackhorizontal( block,block,block,block)
block = stackvertical( block,block,block,block) .pointresize(32,32) .binarize(upper=false)
block = stackhorizontal( block,block,block,block,block,block,block,block)
block = stackvertical( block,block,block,block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block,block)
block = block .crop(0,0,ox,oy)
block = (uv!=3) ? block
\ : YtoUV(block.crop(0,0,ox/2,oy/2),block.crop(0,0,ox/2,oy/2),block)
block = block.trim(1,1) .loop(framecount(clp))


# create normal deblocking (for block borders) and strong deblocking (for block interiour)
normal = clp.deblock(quant=quant1,aOffset=aOff1,bOffset=bOff1)
strong = clp.deblock(quant=quant2,aOffset=aOff2,bOffset=bOff2)

# build difference maps of both
normalD = yv12lutxy(clp,normal,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)
strongD = yv12lutxy(clp,strong,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)

# separate border values of the difference maps, and set the interiours to '128'
strongD2 = yv12lutxy(StrongD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)
normalD2 = yv12lutxy(normalD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)

# interpolate the border values over the whole block: DCTFilter can do it. (Kiss to Tom Barry!)
# (Note: this is not fully accurate, but a reasonable approximation.)
strongD3 = strongD2.yv12lut("x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +",U=uv,V=uv).dctfilter(1.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0)# .yv12lut("x 128 - 2 / 128 +")

# apply compensation from "normal" deblocking to the borders of the full-block-compensations calculated
# from "strong" deblocking ...
strongD4 = yv12lutxy(strongD3,normalD2,"y 128 = x y ?","y 128 = x y ?","y 128 = x y ?",U=uv,V=uv)

# ... and apply it.
deblocked= yv12lutxy(clp,strongD4,"x y 128 - -","x y 128 - -","x y 128 - -",U=uv,V=uv)

# simple decisions how to treat chroma
deblocked = (uv<0) ? deblocked.mergechroma(strong) : uv<2 ? deblocked.mergechroma(normal) : deblocked

deblocked
return( last )
}

[unskinnyboy]

OK, copy/paste and replace your version with this:

function Deblock_QED ( clip clp, int "quant1", int "quant2",
\ int "aOff1", int "bOff1", int "aOff2", int "bOff2", int "uv" )
{
quant1 = default( quant1, 20 )
quant2 = default( quant2, 40 )

aOff1 = default( aOff1, quant1/2 ) # I've no clue if these are clever values or not!
bOff1 = default( bOff1, quant1/4 ) # So:
aOff2 = default( aOff2, quant1/4 ) # Also try all these 4 values @ 0 (zero),
bOff2 = default( bOff2, quant1/8 ) # and quant1=30, quant2=40~45 instead.
uv = default( uv, 1 ) # u=3 -> use proposed method for chroma deblocking
# u=2 -> no chroma deblocking at all (fastest method)
ox = clp.width() # u=1|-1 -> directly use chroma debl. from the normal|strong deblock()
oy = clp.height()

# With avisynth scripting, there is no information available about the position of the currently
# processed pixel ... there simply is no such thing like an "actual" processed pixel.
# So first I've to build up a grid covering the transitions between all 8x8 blocks,
# and make some LUTmania with it later. Yes, this is cumbersome.

block = blankclip(clp,width=6*4,height=6*4,color=$000000).addborders(4,4,4,4,color=$FFFFFF)
block = stackhorizontal( block,block,block,block)
block = stackvertical( block,block,block,block) .pointresize(32,32) .binarize(upper=false)
block = stackhorizontal( block,block,block,block,block,block,block,block)
block = stackvertical( block,block,block,block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block,block)
block = block .crop(0,0,ox,oy)
block = (uv!=3) ? block
\ : YtoUV(block.crop(0,0,ox/2,oy/2),block.crop(0,0,ox/2,oy/2),block)
block = block.trim(1,1) .loop(framecount(clp))


# create normal deblocking (for block borders) and strong deblocking (for block interiour)
normal = clp.deblock(quant=quant1,aOffset=aOff1,bOffset=bOff1)
strong = clp.deblock(quant=quant2,aOffset=aOff2,bOffset=bOff2)

# build difference maps of both
normalD = yv12lutxy(clp,normal,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)
strongD = yv12lutxy(clp,strong,"x y - 128 +","x y - 128 +","x y - 128 +",U=uv,V=uv)

# separate border values of the difference maps, and set the interiours to '128'
strongD2 = yv12lutxy(StrongD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)
normalD2 = yv12lutxy(normalD,block,"y 255 = x 128 ?","y 255 = x 128 ?","y 255 = x 128 ?",U=uv,V=uv)

# interpolate the border values over the whole block: DCTFilter can do it. (Kiss to Tom Barry!)
# (Note: this is not fully accurate, but a reasonable approximation.)
strongD3 = strongD2.yv12lut("x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +",U=uv,V=uv).dctfilter(1.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0)# .yv12lut("x 128 - 2 / 128 +")

# apply compensation from "normal" deblocking to the borders of the full-block-compensations calculated
# from "strong" deblocking ...
strongD4 = yv12lutxy(strongD3,normalD2,"y 128 = x y ?","y 128 = x y ?","y 128 = x y ?",U=uv,V=uv)

# ... and apply it.
deblocked= yv12lutxy(clp,strongD4,"x y 128 - -","x y 128 - -","x y 128 - -",U=uv,V=uv)

# simple decisions how to treat chroma
deblocked = (uv<0) ? deblocked.mergechroma(strong) : uv<2 ? deblocked.mergechroma(normal) : deblocked

deblocked
return( last )
}

[mrkazador]

I actually did try that before. I copied and pasted what you posted and got this error message....
http://img211.imageshack.us/img211/807/afc3.jpg
Which would be this line here:
strongD3 = strongD2.yv12lut("x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +","x 128 - 1.01 * 128 +",U=uv,V=uv).dctfilter(1.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0)# .yv12lut("x 128 - 2 / 128 +")

Here is my script if you need it:


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\dgdecode.dll")
import("C:\Program Files\AviSynth 2.5\plugins\deblock_qed.avs")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TIVTC.dll")
mpeg2source("C:\Documents and Settings\Owner\Desktop\a.d2v")

tfm().tdecimate()
#crop(16,16,-16,-16)
#addborders(0,140,0,140)
lanczosresize(1920,1080)
Deblock_QED(quant1=18,aOff1=16,quant2=28,aOff2=11)
undot()
converttoyuy2()

[Didée]

The code as posted by unskinnyboy is correct. I actually tried it on some HD footage, it works. Can't tell what's wrong on your side ... try disabling all filters before the deblock-filter, and also re-check the corrected script with smaller footage. At some point the Access Violation should be triggered.

BTW, you should not resize before a deblocker! If needed, only crop MOD16 or MOD8 borders beforehand.
If you resize, then the borders of the blocks are no more lying in those positions where the deblocker is assuming that they are!

[mrkazador]

Thanks for that tip im still learning. Seems like im still having problems though......

I erased everything in my Avisynth plugins folder. I installed just what was need for deblock_qed:
dgdecode.dll
deblock_qed()
Masktools.dll File version 1.5.1.0
DctFilter.dll

Changed my script to:

import("C:\Program Files\AviSynth 2.5\plugins\deblock_qed.avs")
mpeg2source("C:\Documents and Settings\Owner\Desktop\a.d2v")
Deblock_QED()

Same error :(

[foxyshadis]

I bet the access violation is just a yv12lut or dctfilter bug, or an out of memory condition (not unlikely when dealing with 1080p). If you ever actually need 1080p processing, you need to make sure you setmemorymax(bignum) and preferably replace yv12lut with mt_lut.

What's your avisynth version?

[mrkazador]

2.60 build Aug 11 2006

[foxyshadis]

That'd explain it. 2.60 is barely even alpha quality, it's not compatible with current avisynth filters; it's more a preview for developers who will evenually need to make their filters compatible. It's pointed out very clearly in the 2.6 thread! You must use a 2.5.x version.

[mrkazador]

Changed to version 2.57 Sep 16 2006
Same problem, I even tried what you suggested.
I also tried deblock_qed_mt2, same thing.

Edit:
Just discovered some new information. Playing the avs script in zoom player i get this error....
**Sorry, Dctfilter test version needs size multiples of 16x16.**

Edit:
Seems like dctfilter does not like my 1920x1080 file. If i resize before deblock_qed and set lanzos to (1920,1088) deblock works..........

[foxyshadis]

Didn't you read what Didée said?

[mrkazador]

I did read it but I guess I didnt understand him correctly. Now i do and it works nicely using crop or addborders.
Thank you guys very much for your help.

[frednerk]

Oh dear. I thought I was running the latest version of Deblock_QED from Clouded's http://www.avisynth.org/mediawiki/wiki/Deblock_QED (the MT2 version)... it seems different to both versions above specifically in the stackings ... which is the latest, I wonder, or is it a "special" for this thread ? Thanks in advance.

#http://www.avisynth.org/mediawiki/wiki/Deblock_QED (I added MMT2 to function name)
function Deblock_QED_MT2 ( clip clp, int "quant1", int "quant2",
\ int "aOff1", int "bOff1", int "aOff2", int "bOff2", int "uv" )
{
quant1 = default( quant1, 20 )
quant2 = default( quant2, 40 )

aOff1 = default( aOff1, quant1/2 ) # I've no clue if these are clever values or not!
bOff1 = default( bOff1, quant1/4 ) # So:
aOff2 = default( aOff2, quant1/4 ) # Also try all these 4 values @ 0 (zero),
bOff2 = default( bOff2, quant1/8 ) # and quant1=30, quant2=40~45 instead.
uv = default( uv, 3 ) # u=3 -> use proposed method for chroma deblocking
# u=2 -> no chroma deblocking at all (fastest method)
ox = clp.width() # u=1|-1 -> directly use chroma debl. from the normal|strong deblock()
oy = clp.height()

# With avisynth scripting, there is no information available about the position of the currently
# processed pixel ... there simply is no such thing like an "actual" processed pixel.
# So first I've to build up a grid covering the transitions between all 8x8 blocks,
# and make some LUTmania with it later. Yes, this is cumbersome.

block = blankclip(clp,width=6*4,height=6*4,color=$000000).addborders(4,4,4,4,color=$FFFFFF)
block = stackhorizontal( block,block,block,block)
block = stackvertical( block,block,block,block) .pointresize(32,32) .mt_binarize(upper=false)
block = stackhorizontal( block,block,block,block,block,block,block,block)
block = stackvertical( block,block,block,block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block,block)
block = stackhorizontal( block,block,block)
block = stackvertical( block,block)
#return clp.subtitle(string(block.width)+"x"+string(block.height))
block = block .crop(0,0,ox,oy)
block = (uv!=3) ? block
\ : YtoUV(block.crop(0,0,ox/2,oy/2),block.crop(0,0,ox/2,oy/2),block)
block = block.trim(1,1) .loop(framecount(clp))


# create normal deblocking (for block borders) and strong deblocking (for block interiour)
normal = clp.deblock(quant=quant1,aOffset=aOff1,bOffset=bOff1)
strong = clp.deblock(quant=quant2,aOffset=aOff2,bOffset=bOff2)

# build difference maps of both
normalD = mt_makediff(clp,normal,chroma=uv>2?"process":"ignore")
strongD = mt_makediff(clp,strong,chroma=uv>2?"process":"ignore")

# separate border values of the difference maps, and set the interiours to '128'
strongD2 = mt_lutxy(StrongD,block,expr="y 255 == x 128 ?",U=uv,V=uv)
normalD2 = mt_lutxy(normalD,block,expr="y 255 == x 128 ?",U=uv,V=uv)

# interpolate the border values over the whole block: DCTFilter can do it. (Kiss to Tom Barry!)
# (Note: this is not fully accurate, but a reasonable approximation.)
strongD3 = strongD2.mt_lut(expr="x 128 - 1.01 * 128 +",U=uv,V=uv).dctfilter(1.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0)# .yv12lut("x 128 - 2 / 128 +")

# apply compensation from "normal" deblocking to the borders of the full-block-compensations calculated
# from "strong" deblocking ...
strongD4 = mt_lutxy(strongD3,normalD2,expr="y 128 == x y ?",U=uv,V=uv)

# ... and apply it.
deblocked= mt_makediff(clp,strongD4,chroma=uv>2?"process":"ignore")

# simple decisions how to treat chroma
deblocked = (uv<0) ? deblocked.mergechroma(strong) : uv<2 ? deblocked.mergechroma(normal) : deblocked

deblocked
return( last )
}

[foxyshadis]

The difference between this and skinny's is mostly cosmetic; it's just that you only need two for the last vertical instead of three, unless processing video with a height of well over 1080p. 2304x1152 should be more than enough for HD processing. (There was a version where it was only performed when required... not really a big deal though, not much of a slowdown. I need to upload a mod8 version though, since DCTFilter doesn't error out like it should.)

[halsboss]

Thanks :) Looking forward to the new mod8 MT version, then. Oh, just to be sure, does that mean this MT2 one posted here already caters for 1080p OK ?

[halsboss]

Hi, any news on the mod8 MT version ?

Also, anyone have a link to explanation of quant1 and quant2 etc ?
(want to end up with a template .AVS with pre-commented-out lines in order of increasing deblock strength, and then just un-comment the one estimated applicable to the source material).

[foxyshadis]

I'd stick this in the function near the top, just for completeness:

Assert(clp.isplanar(),"Deblock_QED: Requires planar YUV (YV12, YV16, YV24, Y8).")
Assert(clp.width() % 8 == 0 && clp.height() % 8 == 0,"Deblock_QED: Requires mod8 video.")


And use a call to this function:
(It's not pretty, wasn't meant for public consumption.)

function makemod(clip c,int "xmod",int "ymod",bool "crop") {
xmod=default(xmod,16)
ymod=default(ymod,xmod)
crop=default(crop,false)
ox=c.width
oy=c.height
w=c.isrgb32() || c.isrgb24() ? 1 : 2
right=c.crop(ox-w,0,w,0)
right=right.stackhorizontal(right,right,right)
right=xmod>w*4 ? right.stackhorizontal(right,right,right) : right
vx=ox % xmod==0 ? c : stackhorizontal(c,right.crop(0,0,xmod-(ox % xmod),0))
bottom=vx.crop(0,oy-w,0,w)
bottom=bottom.stackvertical(bottom,bottom,bottom)
bottom=ymod>w*4 ? bottom.stackvertical(bottom,bottom,bottom) : bottom
vy=oy % ymod==0 ? vx : stackvertical(vx,bottom.crop(0,0,0,ymod-(oy % ymod)))
cropped = c.crop(0,0,int(c.width/xmod)*xmod,int(c.height/ymod)*ymod)
return crop == true ? cropped : vy
}

like:

m=makemod(8)
m.deblockqed().crop(0,0,last.width(),last.height())

Or you can incorporate it directly into DeblockQED, I just prefer the more flexible way.

As for quant1 & 2, well, open up AVSp and make some sliders, try out all different values until you understand. I only say this because I have no idea of the look of the effect, other than being the amount of smoothing/smearing applied. :p