overlapped dct, proof of concept frequency lowpass and interpolator function

[*.mp4 guy]

I made a dct based frequency limiter/expander. The function has variable precision, using 4, 8, 32, or 64 overlapped dct's specified by the "useXdct=true/false" parameters, the highest specified accuracy will be used, if no accuracy is specified 8X accuracy will be used, if all are set to false the lowest accuracy will be used. frequency weighting is by default a sinc function, different frequency weighting can be specified with the dct2-8 parameters, dct2 being the lowest frequncies, dct4 and dct5 being the middle, and dct8 being the highest.

Also included is a simple 4x enlargement function with variable sharpness, and a non-functioning downsizing function.

frequency limiter usage:
dctlimit(dc=1, dct2=1, dct3=1, dct4=1.2, dct5=-0.2, dct6=0, dct7=0, dct8=0, use8dct=true)
dctlimit does not process chroma

upsizer usage:
dctus(sharpness=2)

upsizer2 usage:
wdresize(sharpness=1.95, warp=4, u64dct=false)

The second upsizer function is just a wrapper for the first with some usefull additions, its separated so the other function can be used as a regular interpolater.

Required plugins are dctfilter and average

The upsizer performs about aswell as lanczos, but with less aliasing.

see later post for the function, it almost exceeds the char limit

[708145]

Do I get it right: Is this like a DCTfilter on steroids?

If so then it's very nice. I was looking for that for a long time.

bis besser,
T0B1A5

[*.mp4 guy]

Yep thats exactly what it is, high precision modes are S L O W.

[foxyshadis]

Needs to come with a warning, "Requires 4 gigs of memory", though. :p Creating a simplified plugin out of it is probably necessary to work with HD resolutions.

Possible idea - what about creating a massively overlapped 8x8 DCT this way, and then doing a 16x16 IDCT on the coefficients, in order to perform the upsizing, instead of all of these machinations with pointresize? It probably wouldn't be too hard to create a filter to do that, along with an offset parameter to get overlapped versions. (Without actually performing the averaging in the filter.)

[*.mp4 guy]

Possible idea - what about creating a massively overlapped 8x8 DCT this way, and then doing a 16x16 IDCT on the coefficients, in order to perform the upsizing, instead of all of these machinations with pointresize?

All of the pointresize calls in the dctlimit function are just used to shift the image without interpolating data, but you were probably refering to the pointresize calls used in the interpolation function?

Needs to come with a warning, "Requires 4 gigs of memory", though. :p Creating a simplified plugin out of it is probably necessary to work with HD resolutions.

Possible idea - what about creating a massively overlapped 8x8 DCT this way, and then doing a 16x16 IDCT on the coefficients, in order to perform the upsizing, instead of all of these machinations with pointresize? It probably wouldn't be too hard to create a filter to do that, along with an offset parameter to get overlapped versions. (Without actually performing the averaging in the filter.)

That would be the "non-half-assed" way to go about this type of resizing :sly:. It would be much more memory efficient if each independant instance of dct filter ulnoaded its memory after it processed its data aswell. Another possible speedup would be using the hct instead of the dct, but I don't know how that would impact quality.

Overlapped 8x8dct to overlapped 16*16 IDCT processing would have less overlapping during idct then during the dct, which might cause artifacts. The way things are now it is either a 2-tap function overlapped to 4 taps (all the benefits of a 4tap function, but no ringing) or a 4 tap function averlapped to 8 taps, using a 16X16 idct would turn it into an 8tap function overlapped to 16taps, which could introduce an unpleasant amount of ringing, but would have amaizing high frequency accuracy.

I can't program myself or I would have started on a plugin already, this method definately looks like it will need some real code to work properly.

[edit] function code has been updated, quality of upsizing improved.

[*.mp4 guy]

see later posts for new versions

[Fizick]

I do not know any fast (MMX) DCT 16x16 code.
But I develop dct3dfilter (not finished).

[*.mp4 guy]

More info please.

Does dct3dfilter do an 8x8x8 3d dct across the horizontal vertical and temporal axis?

Is it a non standard size like 8x8x4?

Hows the speed?

How accurate are the dct and idct algorithms you are using?

How does it deal with improperly temporally lowpassed content?

[Fizick]

1. Now I partially implement mode bt=0, bt=1, bt=3, for size 8x8,
overlap=0, 2, 4.
2.Speed is about twice fft3d.
3.Not very accurate dct,
4.I do not know.
It is under development. Please wait.

IMO Soulhunter and tritical have similar filters :)

[foxyshadis]

It would be much more memory efficient if each independant instance of dct filter ulnoaded its memory after it processed its data aswell.

That makes an enormous difference. Enjoy (http://foxyshadis.slightlydark.com/random/DctFilter.zip).

I also added a chroma=-1 parameter so that it won't bother doing all that, since it's grey-only currently. 0 also works, but I'm reserving that for "copy" eventually (current behavior is "trash it"). Also enabled the dormant SSE2 fdct; it hasn't crashed on me so far, though the idct was buggy.

btw, you could put the mergechroma immediately before the return, instead of multiple times after each average; not much of a speedup but something. And I think you mixed up the rep logic.

Prettied up version:

# DCTlimit, by *.mp4 guy
#

function dctlimit(clip c, float "dct2", float "dct3", float "dct4", float "dct5", float "dct6", float "dct7", float "dct8", float "dc", bool "use8dct", bool "use32dct", bool "use64dct", bool "use4dct"){

dc = default(dc, 1)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 0)
dct6 = default(dct6, 0)
dct7 = default(dct7, 0)
dct8 = default(dct8, 0)

use4dct = default(use4dct, false)
use8dct = default(use8dct, true)
use32dct = default(use32dct, false)
use64dct = default(use64dct, false)

w = c.width
h = c.height

sourcey = c
source = sourcey
cropc = sourcey
#crop0 = sourcey.dctfilter(dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8,chroma=-1)

##################################################################-dct16-1
crop1 = source.dfilter(-1, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop2 = source.dfilter(-1, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop3 = source.dfilter(-1, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop4 = source.dfilter(-1, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop5 = source.dfilter(-1, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop6 = source.dfilter(-1, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop7 = source.dfilter(-1, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop8 = source.dfilter(-1, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop9 = source.dfilter(-2, -0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop10 = source.dfilter(-2, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop11 = source.dfilter(-2, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop12 = source.dfilter(-2, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop13 = source.dfilter(-2, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop14 = source.dfilter(-2, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop15 = source.dfilter(-2, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop16 = source.dfilter(-2, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

##################################################################-dct32-2
crop17 = source.dfilter(-3, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop18 = source.dfilter(-3, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop19 = source.dfilter(-3, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop20 = source.dfilter(-3, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop21 = source.dfilter(-3, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop22 = source.dfilter(-3, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop23 = source.dfilter(-3, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop24 = source.dfilter(-3, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop25 = source.dfilter(-4, -0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop26 = source.dfilter(-4, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop27 = source.dfilter(-4, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop28 = source.dfilter(-4, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop29 = source.dfilter(-4, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop30 = source.dfilter(-4, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop31 = source.dfilter(-4, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop32 = source.dfilter(-4, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

##################################################################-dct48-3
crop33 = source.dfilter(-5, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop34 = source.dfilter(-5, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop35 = source.dfilter(-5, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop36 = source.dfilter(-5, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop37 = source.dfilter(-5, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop38 = source.dfilter(-5, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop39 = source.dfilter(-5, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop40 = source.dfilter(-5, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop41 = source.dfilter(-4, -0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop42 = source.dfilter(-6, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop43 = source.dfilter(-6, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop44 = source.dfilter(-6, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop45 = source.dfilter(-6, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop46 = source.dfilter(-6, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop47 = source.dfilter(-6, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop48 = source.dfilter(-6, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

##################################################################-dct64-4
crop49 = source.dfilter(-7, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop50 = source.dfilter(-7, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop51 = source.dfilter(-7, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop52 = source.dfilter(-7, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop53 = source.dfilter(-7, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop54 = source.dfilter(-7, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop55 = source.dfilter(-7, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop56 = source.dfilter(-7, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop57 = source.dfilter(-0, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop58 = source.dfilter(-0, -1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop59 = source.dfilter(-0, -2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop60 = source.dfilter(-0, -3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop61 = source.dfilter(-0, -4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop62 = source.dfilter(-0, -5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop63 = source.dfilter(-0, -6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
crop64 = source.dfilter(-0, -7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

###################################
alignc = cropc
###################################


##################################################################-align16-1
align1 = crop1 .pointResize(W, H, 1, 0, W, H)
align2 = crop2 .pointResize(W, H, 1, 1, W, H)
align3 = crop3 .pointResize(W, H, 1, 2, W, H)
align4 = crop4 .pointResize(W, H, 1, 3, W, H)
align5 = crop5 .pointResize(W, H, 1, 4, W, H)
align6 = crop6 .pointResize(W, H, 1, 5, W, H)
align7 = crop7 .pointResize(W, H, 1, 6, W, H)
align8 = crop8 .pointResize(W, H, 1, 7, W, H)
align9 = crop9 .pointResize(W, H, 2, 0, W, H)
align10 = crop10.pointResize(W, H, 2, 1, W, H)
align11 = crop11.pointResize(W, H, 2, 2, W, H)
align12 = crop12.pointResize(W, H, 2, 3, W, H)
align13 = crop13.pointResize(W, H, 2, 4, W, H)
align14 = crop14.pointResize(W, H, 2, 5, W, H)
align15 = crop15.pointResize(W, H, 2, 6, W, H)
align16 = crop16.pointResize(W, H, 2, 7, W, H)

##################################################################-align32-2
align17 = crop17.pointResize(W, H, 3, 0, W, H)
align18 = crop18.pointResize(W, H, 3, 1, W, H)
align19 = crop19.pointResize(W, H, 3, 2, W, H)
align20 = crop20.pointResize(W, H, 3, 3, W, H)
align21 = crop21.pointResize(W, H, 3, 4, W, H)
align22 = crop22.pointResize(W, H, 3, 5, W, H)
align23 = crop23.pointResize(W, H, 3, 6, W, H)
align24 = crop24.pointResize(W, H, 3, 7, W, H)
align25 = crop25.pointResize(W, H, 4, 0, W, H)
align26 = crop26.pointResize(W, H, 4, 1, W, H)
align27 = crop27.pointResize(W, H, 4, 2, W, H)
align28 = crop28.pointResize(W, H, 4, 3, W, H)
align29 = crop29.pointResize(W, H, 4, 4, W, H)
align30 = crop30.pointResize(W, H, 4, 5, W, H)
align31 = crop31.pointResize(W, H, 4, 6, W, H)
align32 = crop32.pointResize(W, H, 4, 7, W, H)

##################################################################-align48-3
align33 = crop33.pointResize(W, H, 5, 0, W, H)
align34 = crop34.pointResize(W, H, 5, 1, W, H)
align35 = crop35.pointResize(W, H, 5, 2, W, H)
align36 = crop36.pointResize(W, H, 5, 3, W, H)
align37 = crop37.pointResize(W, H, 5, 4, W, H)
align38 = crop38.pointResize(W, H, 5, 5, W, H)
align39 = crop39.pointResize(W, H, 5, 6, W, H)
align40 = crop40.pointResize(W, H, 5, 7, W, H)
align41 = crop41.pointResize(W, H, 6, 0, W, H)
align42 = crop42.pointResize(W, H, 6, 1, W, H)
align43 = crop43.pointResize(W, H, 6, 2, W, H)
align44 = crop44.pointResize(W, H, 6, 3, W, H)
align45 = crop45.pointResize(W, H, 6, 4, W, H)
align46 = crop46.pointResize(W, H, 6, 5, W, H)
align47 = crop47.pointResize(W, H, 6, 6, W, H)
align48 = crop48.pointResize(W, H, 6, 7, W, H)

##################################################################-align64-4
align49 = crop49.pointResize(W, H, 7, 0, W, H)
align50 = crop50.pointResize(W, H, 7, 1, W, H)
align51 = crop51.pointResize(W, H, 7, 2, W, H)
align52 = crop52.pointResize(W, H, 7, 3, W, H)
align53 = crop53.pointResize(W, H, 7, 4, W, H)
align54 = crop54.pointResize(W, H, 7, 5, W, H)
align55 = crop55.pointResize(W, H, 7, 6, W, H)
align56 = crop56.pointResize(W, H, 7, 7, W, H)
align57 = crop57#.pointResize(W, H, 0, 0, W, H)
align58 = crop58.pointResize(W, H, 0, 1, W, H)
align59 = crop59.pointResize(W, H, 0, 2, W, H)
align60 = crop60.pointResize(W, H, 0, 3, W, H)
align61 = crop61.pointResize(W, H, 0, 4, W, H)
align62 = crop62.pointResize(W, H, 0, 5, W, H)
align63 = crop63.pointResize(W, H, 0, 6, W, H)
align64 = crop64.pointResize(W, H, 0, 7, W, H)

dct4_1 = average(align56, 0.25, align38, 0.25, align20, 0.25, align2, 0.25)
dct8_1 = average(align57, 0.125, align56, 0.125, align38, 0.125, align47, 0.125, align20, 0.125, align29, 0.125, align2, 0.125, align11, 0.125)
dct16_1 = average(align1, 0.0625, align2, 0.0625, align3, 0.0625, align4, 0.0625, align5, 0.0625, align6, 0.0625, align7, 0.0625, align8, 0.0625, align9, 0.0625, align10, 0.0625, align11, 0.0625, align12, 0.0625, align13, 0.0625, align14, 0.0625, align15, 0.0625, align16, 0.0625)
dct32_2 = average(align17, 0.0625, align18, 0.0625, align19, 0.0625, align20, 0.0625, align21, 0.0625, align22, 0.0625, align23, 0.0625, align24, 0.0625, align25, 0.0625, align26, 0.0625, align27, 0.0625, align28, 0.0625, align29, 0.0625, align30, 0.0625, align31, 0.0625, align32, 0.0625)
dct48_3 = average(align33, 0.0625, align34, 0.0625, align35, 0.0625, align36, 0.0625, align37, 0.0625, align38, 0.0625, align39, 0.0625, align40, 0.0625, align41, 0.0625, align42, 0.0625, align43, 0.0625, align44, 0.0625, align45, 0.0625, align46, 0.0625, align47, 0.0625, align48, 0.0625)
dct64_4 = average(align49, 0.0625, align50, 0.0625, align51, 0.0625, align52, 0.0625, align53, 0.0625, align54, 0.0625, align55, 0.0625, align56, 0.0625, align57, 0.0625, align58, 0.0625, align59, 0.0625, align60, 0.0625, align61, 0.0625, align62, 0.0625, align63, 0.0625, align64, 0.0625)

align57
use4dct ? dct4_1 : last
use8dct ? dct8_1 : last
use32dct ? average(dct16_1, 0.5, dct32_2, 0.5) : last
use64dct ? average(dct16_1, 0.25, dct48_3, 0.25, dct32_2, 0.25, dct64_4, 0.25) : last
mergechroma(alignc, 1)

return(last)
}

function m(int r, float x) {return(x<16?16:int(round(x/float(r))*r))}



Function DCTus(clip clp, float "sharpness", bool "u64dct"){
OX = m(16,Clp.Width)
OY = m(16,Clp.Height)
dx = ox - clp.width
dy = oy - clp.height
sharpness = default(sharpness, 2)
u64dct = default(u64dct, false)
luma = clp.pointresize(Clp.Width*4,Clp.Height*4).addborders(dx*4, dy*4, 0, 0).dctlimit(dct2=sharpness, dct3=0, dct4=0, dct5=0, use64dct=u64dct, use8dct=true).crop(dx*4, dy*4, 0, 0)
luma.mergechroma(lanczosresize(clp,Clp.Width*4 ,Clp.Height*4), 1)
return(last)
}

Function DCTds(clip clp,int destx, int desty, float "sharpness"){
sharpness = default(sharpness,1)
OX = m(16,Clp.Width)
OY = m(16,Clp.Height)
dx = ox - clp.width
dy = oy - clp.height
(clp.width < destx*2) ? dx*2 : dx
(clp.width < destx*2) ? dy*2 : dy
clp
(width < destx*2) ? dctus(sharpness) : last
(width < destx*2) ? dctds(destx=destx, desty=desty, sharpness=sharpness) : bilinearresize(destx,desty).addborders(dx, dy, 0, 0).dctlimit(dct3=0.75+(sharpness/4),dct4=sharpness, dct5=0).crop(dx, dy, 0, 0).pointresize(destx,desty)
return last
}

function wdresize(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep"){
sharpness = default(sharpness, 2)
u64dct = default(u64dct, false)
warp = default(warp, 4)
thresh = default(thresh, 50)
rep = default(rep, 1)

clp.dctus(sharpness=sharpness, u64dct=u64dct).DeHalo_alpha(ss=1, lowsens=30, highsens=30, rx=2, ry=2)

w = last.width
h = last.height
pointresize(w/2, h/2)
(rep>= 2) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(rep>= 2) ? last.DeHalo_alpha(ss=1, lowsens=15, highsens=15, rx=2, ry=2) : last

(rep>= 3) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(rep>= 3) ? last.DeHalo_alpha(ss=1, lowsens=20, highsens=20, rx=2, ry=2) : last

awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
return(last)
}

function dfilter(clip c, int offx, int offy, float dc, float dct2, float dct3, float dct4, float dct5, float dct6, float dct7, float dct8) {
c.PointResize(c.width,c.height,offx,offy,c.width,c.height).dctfilter(dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8,chroma=-1)
}

[*.mp4 guy]

Thanks for the help (!) dctfilter doesn't crash avisynth on frames larger then 352*192 now :D.

The script cleanup is quite nice aswell. One question though, how do you set which dct dctfilter uses?

Minor update to resizing function, defaults improved, it is now possible to set the initial taps, non mod 16 imput resolutions now work, rep code had a minor cleanup and halo removal strength is now a variable:
function wdresize(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep", int "antihalo", int "taps", int "dehalo"){
sharpness = default(sharpness, 2)
u64dct = default(u64dct, true)
warp = default(warp, 4)
thresh = default(thresh, 50)
rep = default(rep, 1)
antihalo = default(antihalo, 150)
taps = default(taps, 4)
dehalo = default(dehalo, rep)

cw = clp.width
ch = clp.height

(taps <=3) ? clp.dctus(sharpness=sharpness, u64dct=u64dct).DeHalo_alpha(ss=1, lowsens=antihalo, highsens=antihalo, rx=2, ry=2) : last

(taps >=3) ? clp.pointresize(cw*2, ch*2).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last

(dehalo >=1) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

w = last.width
h = last.height

(taps <=3) ? pointresize(w/2, h/2) : last

OX = m(16,Clp.Width)
OY = m(16,Clp.Height)
dx = ox - clp.width
dy = oy - clp.height

addborders(dx, dy, 0, 0)

(rep>=2) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=2) ? last.DeHalo_alpha(ss=1, lowsens=antihalo, highsens=antihalo, rx=2, ry=2) : last

(rep>=3) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=3) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=4) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=4) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=5) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=5) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=6) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=6) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=7) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=7) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=8) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=0) : last
(dehalo >=8) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last


crop(dx, dy, 0, 0)

awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)

mergechroma(lanczosresize(clp,clp.Width*2 ,clp.Height*2), 1)

return(last)}
on the todo list -
-intelligent masking to tone down halos and ringing (this is going to be really hard, but worth it if I figure something out)
-chroma processing
-investigate the effects of dct-idct precision on visible artifacts after multiple frequency seperation repetitions
-tone down the flattening caused by dehaloing. (see 1)
-look into 16*16 idct speed versus 8*8 idct speed.


[edit] upsizing function updated again, now suppots up to 8 reps (which very surprisingly actually looks good on some images, though I'm beginning to suspect that the reason lenna image is used so frequently in digital image processing is that it is perfectly lowpassed... it makes every image processing algorithm look like a miracle worker)

[foxyshadis]

Thanks for the help (!) dctfilter doesn't crash avisynth on frames larger then 352*192 now :D.
Cool!

The script cleanup is quite nice aswell. One question though, how do you set which dct dctfilter uses?
It sets it by the processor it detects right now. There isn't any floating-point dct, only mmx/sse2 dct and mmx/isse idct, so they all have the same (lousy) precision. If I can find a way to get a float in there (like mplayer's) I'll make it selectable.

[Si]

Someones got to ask and it may as well be me :)
I made a dct based frequency limiter/expander. The function has variable precision, using 4, 8, 32, or 64 overlapped dct's specified by the "useXdct=true/false" parameters, the highest specified accuracy will be used, if no accuracy is specified 8X accuracy will be used, if all are set.....

And what's one of them then ?

regards

Simon

[*.mp4 guy]

Someones got to ask and it may as well be me :)


And what's one of them then ?

regards

Simon
It's a highly precise bluring/sharpening/line detection algorithm that can be used to remove artifacts when upsizing and downsizing by correctly removing frequencies that cause aliasing ringing and haloing, without removing frequencies that are actually detail.

It sets it by the processor it detects right now. There isn't any floating-point dct, only mmx/sse2 dct and mmx/isse idct, so they all have the same (lousy) precision. If I can find a way to get a float in there (like mplayer's) I'll make it selectable.

That might explain something, is the mmx sse2 dct more precise then the mmx isse one? I thought the results I got were uncharacteristically low on ringing after I updated dctfilter and a difference in precision would explain it.


I've been thinking about sampling theory, and the theoretically impossible sinc function, and I realised that not only would you need an infinitely precise sinc function to get perfect frequency elimination, you would also need an infinitely precise colorspace and sampling depth, IE you would need an infinte resolution image with an infinite colorspace just to store the results of the perfect sinc function. So given that it is both impossible to create a perfect sinc function and to adequately sample its results, shouldn't it then be possible to use a sufficiently precise sinc function so that the error it produces is lower then the accuracy loss of your destination resolution and colorspace, thereby giving you results that are as good as the perfect sinc functions results would be had they been subsampled to the resolution and colorspace you are using. Of couse in all likely hood such a sufficiently precise sinc function would be so slow that it would be completely useless. Given that a perfect dct transform is impossible, I think its possible that a sufficiently precise dct could impersonate the perfect sinc function with an adequately imprecise output resolution and colorspace.

[Didée]

The concept sounds interesting. But somehow I have some issues ... perhaps I'm missing something?

1. In whichever way I juggle around with true/false for all the useXdct, the result is always exactly the same?

2. Looking at what a dctlimit() call does to a sample frame:

original: http://img123.imageshack.us/img123/2972/dctlimitoriglm5.th.png (http://img123.imageshack.us/my.php?image=dctlimitoriglm5.png)

dctlimit: http://img293.imageshack.us/img293/706/dctlimit64dctcb5.th.png (http://img293.imageshack.us/my.php?image=dctlimit64dctcb5.png)


Well, there's definetly an effect, but I see no use for what it's giving to me ...

Is the function really doing what it's supposed to do? Am I missing something obvious?

[*.mp4 guy]

The concept sounds interesting. But somehow I have some issues ... perhaps I'm missing something?

1. In whichever way I juggle around with true/false for all the useXdct, the result is always exactly the same?

2. Looking at what a dctlimit() call does to a sample frame:

original: http://img123.imageshack.us/img123/2972/dctlimitoriglm5.th.png (http://img123.imageshack.us/my.php?image=dctlimitoriglm5.png)

dctlimit: http://img293.imageshack.us/img293/706/dctlimit64dctcb5.th.png (http://img293.imageshack.us/my.php?image=dctlimit64dctcb5.png)


Well, there's definetly an effect, but I see no use for what it's giving to me ...

Is the function really doing what it's supposed to do? Am I missing something obvious?

Are you
1-trying to resize the image
2-trying to lowpass
3-trying to enhance detail

If you are trying to do 1 you need to use the wdresize() function instead of dctlimit, wdresize is SLOW.

If you are trying to do #2, you just have, but you need to remove more frequencoes above the nyquist limit, try

dctlimit(use64dct=true, dct2=1, dct3=2, dct4=2, dct5=0)
dctlimit(use64dct=true, dct2=1, dct3=2, dct4=0.5, dct5=2)

dctlimit(use64dct=true, dct2=1, dct3=2, dct4=2, dct5=0)
dctlimit(use64dct=true, dct2=1, dct3=2, dct4=0.5, dct5=2)

repetition refines the frequency seperation, giving more accurate results, but also causes errors by the dct-idct algorithms to be amplified.

if you are trying to do # 3 you would call dctlimit like this

dctlimit(use64dct=true, dct2=1, dct3=1, dct4=1, dct5=1, dct6=1.5, dct7=2, dct8=2)

I havent extensively tested using dctlimit for accurate frequency sharpening.


the "usxdct" parameter adds accuracy, when all you are doing is running a single operation of dctlimit the difference between use8dct and use64dct will be impossible to see without enhancement of the image.

[CAFxX]

A little ago I did something similar, i.e. performing a fft on the whole image and then padding/cropping to reach the desired size.
I used the FFTW library and I got this (video is the second bravia commercial, 1280x720 downsized to 320x180 via spline36 and then upconverted with my filter (and spline36 as a reference); luma only atm; spline first, fft second):
frame 370:
http://img214.imageshack.us/img214/4466/370sye1.th.png (http://img214.imageshack.us/img214/4466/370sye1.png)
http://img214.imageshack.us/img214/4893/370fqt2.th.png (http://img214.imageshack.us/img214/4893/370fqt2.png)
frame 817:
http://img214.imageshack.us/img214/3529/817sza6.th.png (http://img214.imageshack.us/img214/3529/817sza6.png)
http://img214.imageshack.us/img214/5793/817fmi4.th.png (http://img214.imageshack.us/img214/5793/817fmi4.png)
I suppose the ringing effect near the borders is due to the limited precision but I actually can't guarantee that my code is bugless.
atm it can't do downsizing, but in matlab downsizing using this method proved interesting.
the filter is quite slow (15-20fps on my athlon64 3200), and it takes a while to load (fftw needs some time to prepare the ffts).
obviously (granted that the above mentioned ringing effect is due to limited precision) a overlapping window approach should resolve the issue. (and prove faster: fftw works in double precision... a highly tuned costant size integer/fixed-point algorithm will surely go much faster: btw, if anyone can point me towards an integer fft i may try to use that instead)

edit: found IPPs: will try them later

[*.mp4 guy]

Overlapping should always increase precision for frequency transforms that are not odd, that is overlapping will always improve precision when used with a dct or an fft (they only work on even blocks of data) however the optimal way to avoid problems with any frequency transform is to only use the center pixel, as the center pixel has the least error, so for the most precise 8x8 dct-idct data you would need to perform 9 dct transforms per group of four pixels, avergaing the results and using only the center four pixels of the average after idct, using a different group of 9 transforms for each group of four pixels in the entire image, each group sharing 8 transforms with its neighbors (1 left, 1 up, 1 uperleft, etc.), this would give the most precise reconstruction (better then 64x overlapping). The same aplies to the fft, so the best ways to improve results are to 1-increase transform size, 2-use sufficient overlapping and pixel binning 3-use a precise transform and inverse transform.

After some additional research it turns out that the optimal frequency domain transform is the KLT(Karhunen-Loève transform), which is better then the dct, which is better then the fft. The klt is the most optimal linear transform, for the same reason the dct is better then the fft, that is it keeps the most important data only at the lowest frequencies, whereas with the dct and fft some important information is kept at higher frequencies then unimportant information, the fft has low frequency seperation, whereas the dct has high frequency seperation, and the klt has perfect frequency seperation. A perfectly resmapled image could be made using a sufficiently precise aproximation of a klt and a sufficiently imprecise colorspace and destination resolution, the klt would have to use variable transform sizes, the interior of the image would use a transform the size of the entire image, as the transform moved twards the borders of the image the transform size would have to be reduced to stay within the image boundaries.

[Didée]

@ *.mp4 guy

Thanks for all the explanations, but I already know the drill. ;)

You didn't get my point ... I was wondering if the function is working correctly.
Answer is: it does, and it does not.
And if you would have looked at the sample, you should have noted that something is definetly going wrong there ...

Solution:
Like it is posted, dctlimit() will work correctly only with Avisynth v2.57.
But when used with Avisynth v2.56, the result is a joke ... because of the resizer bug of AS 2.56.

I added the needed workaround, & put up the modified script here (http://home.arcor.de/dhanselmann/_stuff/dctlimit.avs).
It's not needed for v.257, but it doesn't hurt, either.

(No need to discuss the "why still use 2.56???" question ... it is still used, and there the original function "silently" gives a b0rked result. Quite sneaky if one don't know about that ... it seems to work, but the result is crap, and people might wonder what it is all about ...
Better to play safe, for the time being.)

[*.mp4 guy]

The results were bad on that sample, but I wasn't sure why, I figured it was just bad settings/ a tough source, thanks for the fix/workaround.

Also, does anyone know where I can get more information on the KLT/PCA transform? Wikipedias explanation was nice, but I would like more information about practical uses.

[foxyshadis]

I did put up a version of dctfilter that allows you to choose dct, if you want to compare: fdct=1/3, idct=1/2/3, 1=mmx, 2=isse, 3=sse2, 0=int but I'm still trying to get that one to work. The 16-bit int method is going to be excruciatingly slow, though.

I have no idea if the simd versions really differ, but it can't hurt to be sure. The math and assembly hurts my head.

Edit: Made another change, doesn't seem to have affected allocation strategies at all, still searching.

Ah, it does seem that the pointresize is where all the memory is getting sucked into now, I guess you'd need a custom avisynth. Actually, I guess a dll with just a custom version of that would work.

[CAFxX]

I rewrote the code of the filter in order to use the IPPs.
It's nowhere near being optimized (nor cpu-wise, nor memory-wise) or easily usable (only "multiplicative" upsizing ATM).
As far as the name is concerned, it will probably change in future.

sincresize(clip, int factor, int window)
clip - input clip: planar, mod 2^window
factor - scaling parameter: >= 2
window - size of first DCT: >= 2

the filter works like this:
for every (square) block of size 2^window of the source frame
- perform a forward DCT
- pad the above DCT in an empty block of size (2^window)*factor
- perform the inverse DCT on the padded block
- extract from the middle of the block a window of size (2^window)*factor/2 and copy it to the destination frame (this is done to avoid blocking)
notes:IPP DFT works on single precision floating point values, near-border regions are handled by replicating the border values.

Download (uncommented source + MSVC binary): http://cafxx.strayorange.com/SincResize.7z
(development and testing platform: Athlon64 3200+, 1GB DDR400, XP SP2, AviSynth 2.57, MSVC8, IPP5.1)

[*.mp4 guy]

@CAFxXI got this error (http://img236.imageshack.us/img236/135/sincresizejg1.png) when I tried to use your plugin. I think when I was testing dctlimit I might have gotten a few bad sectors from heavy disk usage, so I'll run a chkdsk and report back if that changes anything.

@foxyshadis The problem with point resize is that it doesn't unallocate memory after processing?

[CAFxX]

@CAFxXI got this error (http://img236.imageshack.us/img236/135/sincresizejg1.png) when I tried to use your plugin. I think when I was testing dctlimit I might have gotten a few bad sectors from heavy disk usage, so I'll run a chkdsk and report back if that changes anything.
I swear I don't know what is going on there... someone else has that problem?
BTW, i forgot to mention that now the filter also processes the chroma planes.

edit: I also forgot to mention that i haven't added yet the support for named parameters... invoke it like this: sincresize(<factor>, <window>)
in your case would be sincresize(2,2). i'll do that soon, anyway.

[niiyan]

@foxyshadis
I downloaded DctFilter from http://forum.doom9.org/showthread.php?p=951432#post951432, but I can't extract it.
The error message of the extractor says the archive is broken.
Could you check if DctFilter.zip is OK or upload DctFilter.zip again?

[CAFxX]

I packaged a second version of sincresize:
- now accepts named parameters
- parameters default are factor(2) and window(3)
- enabled multi-threading via openmp (I don't have a multicore processor, so actually I can't test if it works)
- removed a couple of useless comments from the sources

download: http://cafxx.strayorange.com/SincResize2.7z

[*.mp4 guy]

I ran chkdsk and I've still got the problem, even after re-downloading the plugin, are you using a non-english characterset in the plugin, that might be what is causing the problems.

[Adub]

@niiyan
Try downloading it again. Sometimes the servers cut things off in the middle of a download so you don't actually finish the entire file.

[foxyshadis]

I think it was my FTP client, actually, which likes to "resume" new uploads at random. I've been uploading new versions (some of which don't crash!) over the course of the day as I'm in and out.

It has an offset codepath right now, that's going to help a lot once it works - no more need for pointresize at all! - but right now it just crashes a lot. I also found out that the original simd dcts are 16-bit, so that makes the 16-bit I was trying to hack in there pretty pointless, meh. Still haven't found any double version, if it's important I can try again some other time.

I guess this is what I get for not understanding much assembly.

Edit: Looking at the co-efficients in the debugger, I see a very easy way to increase the precision by 4 bits, just multiply everything by 16; there's a lot of unused extra precision here, probably at least 6 bits, but definitely 4.

[*.mp4 guy]

On a test image it looks like only 2 more bits of precision are needed to avoid any artifacts using an 8x8dct->16x16idct method, but using the current pointresize then lowpass method it could take up to 7 more bits of precision to avoid errors caused by rounding during frequency scaling =/ I know dgindex has some very precise idcts, but I don't know of any precise dcts. Version 3 of fftw has a dft equivelent to both the dct and idct, and they support variable sizes, but I don't know how precise they are, or if they are fast enough, since performing a dct using a dft is supposed to be slower then using a specialised dct algorithm. I wish I could figure out how to cludge something together in c, but most of dctfilter and sincresize are still unreadable to me.

Also IEEE explore (and all the other ones like them) are annoying, they have 90% of the papers on the dct, the hct, even the klt. meh. From googling it looks like there has been a lot of research on this topic, I can't figure out why there aren't any other implementations.

[CAFxX]

I ran chkdsk and I've still got the problem, even after re-downloading the plugin, are you using a non-english characterset in the plugin, that might be what is causing the problems.
I don't think... in the source file, only pure ASCII chars are used... and all the strings are in english...
Will someone else please confirm wheter the binary is broken or not?
BTW, I had a look at the sources of DGIndex, but I (obviously) found just a 8x8 iDCT: ideally I'd need a 8x8 DCT and 16x16, 32x32 (maybe even 64x64) iDCTs (if downscaling is also needed, then the full array of DCTs and iDCTs is required).
Eventually, if someone can point out a promising book on the matter, I may try to go grab it from one of the libraries of my university.
As a footnote, I found mentions of a former sourceforge project called mmxdct: unfortunately, it looks like it is not available anymore. If you happen to stumble upon a mirror, a cached version, or anything please post the address here.

[*.mp4 guy]

Ehm what I meant, is what charset are you compiling it with, ASCI, Unicode? I"m guessing its a charset problem becuase of the grarbled error avisynth returned, that shouldn't happen for anything but a problem with the charset. I'll let you know if I find any good dct routines.

[edit] this (http://momonga.t.u-tokyo.ac.jp/~ooura/fft.html) might be what your looking for.

[CAFxX]

I did not force Unicode in the MSVC project properties, and in fact if you open the dll with a hex editor and scroll down to the strings you'll see that the chars are not 16bit wide.
Anyway I'll test the dll on another computer. Maybe it's just a missing/wrong version library kind of problem.

[CAFxX]

[edit] this (http://momonga.t.u-tokyo.ac.jp/~ooura/fft.html) might be what your looking for.
Those are plain C double precision floating point routines...

[*.mp4 guy]

Well, they are the only variable size dcts I found, and you aren't doing any overlapping so it shouldn't be that slow. Have you found any other non-mod-8 dct code?

[CAFxX]

uhm... yes and no
IPP can do variable size DCT as well (actually the function is the same... you pass the dimensions of the DCT as parameters), and actually my filter does overlapping...
the filter works like this:
for every (square) block of size 2^window of the source frame
- perform a forward DCT
- pad the above DCT in an empty block of size (2^window)*factor
- perform the inverse DCT on the padded block
- extract from the middle of the block a window of size (2^window)*factor/2 and copy it to the destination frame (this is done to avoid blocking)
maybe it was not that clear, or maybe this is not the kind of overlapping you expected, but it definitely works (precision issues aside)
(anyway, thank you for your interest!)

[foxyshadis]

I really need to keep better track of my mental notes, it would save me so much time debugging later on. So anyway, I got you a version that'll do offsets, but unfortunately several attempts to get higher precision out of it failed. I'm going to have to ask someone who knows mmx/sse to find out why, because there's so much wasted extra room. But at least it takes less memory to do its thing:

Plugin (http://foxyshadis.slightlydark.com/random/DctFilter.zip)


function dctlimit(clip c, float "dct2", float "dct3", float "dct4", float "dct5", float "dct6", float "dct7", float "dct8", float "dc", bool "use8dct", bool "use32dct", bool "use64dct", bool "use4dct"){

dc = default(dc, 1)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 0)
dct6 = default(dct6, 0)
dct7 = default(dct7, 0)
dct8 = default(dct8, 0)

use4dct = default(use4dct, false)
use8dct = default(use8dct, true)
use32dct = default(use32dct, false)
use64dct = default(use64dct, false)

w = c.width
h = c.height

sourcey = c
source = sourcey
cropc = sourcey
#crop0 = sourcey.dctfilter(dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8,chroma=-1)

##################################################################-dct16-1
align1 = source.dfilter( 1, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align2 = source.dfilter( 1, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align3 = source.dfilter( 1, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align4 = source.dfilter( 1, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align5 = source.dfilter( 1, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align6 = source.dfilter( 1, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align7 = source.dfilter( 1, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align8 = source.dfilter( 1, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align9 = source.dfilter( 2, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align10 = source.dfilter( 2, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align11 = source.dfilter( 2, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align12 = source.dfilter( 2, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align13 = source.dfilter( 2, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align14 = source.dfilter( 2, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align15 = source.dfilter( 2, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align16 = source.dfilter( 2, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

################################################################## dct32 2
align17 = source.dfilter( 3, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align18 = source.dfilter( 3, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align19 = source.dfilter( 3, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align20 = source.dfilter( 3, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align21 = source.dfilter( 3, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align22 = source.dfilter( 3, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align23 = source.dfilter( 3, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align24 = source.dfilter( 3, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align25 = source.dfilter( 4, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align26 = source.dfilter( 4, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align27 = source.dfilter( 4, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align28 = source.dfilter( 4, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align29 = source.dfilter( 4, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align30 = source.dfilter( 4, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align31 = source.dfilter( 4, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align32 = source.dfilter( 4, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

################################################################## dct48 3
align33 = source.dfilter( 5, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align34 = source.dfilter( 5, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align35 = source.dfilter( 5, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align36 = source.dfilter( 5, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align37 = source.dfilter( 5, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align38 = source.dfilter( 5, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align39 = source.dfilter( 5, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align40 = source.dfilter( 5, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align41 = source.dfilter( 4, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align42 = source.dfilter( 6, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align43 = source.dfilter( 6, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align44 = source.dfilter( 6, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align45 = source.dfilter( 6, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align46 = source.dfilter( 6, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align47 = source.dfilter( 6, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align48 = source.dfilter( 6, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

################################################################## dct64 4
align49 = source.dfilter( 7, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align50 = source.dfilter( 7, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align51 = source.dfilter( 7, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align52 = source.dfilter( 7, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align53 = source.dfilter( 7, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align54 = source.dfilter( 7, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align55 = source.dfilter( 7, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align56 = source.dfilter( 7, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align57 = source.dfilter( 0, 0, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align58 = source.dfilter( 0, 1, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align59 = source.dfilter( 0, 2, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align60 = source.dfilter( 0, 3, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align61 = source.dfilter( 0, 4, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align62 = source.dfilter( 0, 5, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align63 = source.dfilter( 0, 6, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)
align64 = source.dfilter( 0, 7, dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8)

dct4_1 = average(align56, 0.25, align38, 0.25, align20, 0.25, align2, 0.25)
dct8_1 = average(align57, 0.125, align56, 0.125, align38, 0.125, align47, 0.125, align20, 0.125, align29, 0.125, align2, 0.125, align11, 0.125)
dct16_1 = average(align1, 0.0625, align2, 0.0625, align3, 0.0625, align4, 0.0625, align5, 0.0625, align6, 0.0625, align7, 0.0625, align8, 0.0625, align9, 0.0625, align10, 0.0625, align11, 0.0625, align12, 0.0625, align13, 0.0625, align14, 0.0625, align15, 0.0625, align16, 0.0625)
dct32_2 = average(align17, 0.0625, align18, 0.0625, align19, 0.0625, align20, 0.0625, align21, 0.0625, align22, 0.0625, align23, 0.0625, align24, 0.0625, align25, 0.0625, align26, 0.0625, align27, 0.0625, align28, 0.0625, align29, 0.0625, align30, 0.0625, align31, 0.0625, align32, 0.0625)
dct48_3 = average(align33, 0.0625, align34, 0.0625, align35, 0.0625, align36, 0.0625, align37, 0.0625, align38, 0.0625, align39, 0.0625, align40, 0.0625, align41, 0.0625, align42, 0.0625, align43, 0.0625, align44, 0.0625, align45, 0.0625, align46, 0.0625, align47, 0.0625, align48, 0.0625)
dct64_4 = average(align49, 0.0625, align50, 0.0625, align51, 0.0625, align52, 0.0625, align53, 0.0625, align54, 0.0625, align55, 0.0625, align56, 0.0625, align57, 0.0625, align58, 0.0625, align59, 0.0625, align60, 0.0625, align61, 0.0625, align62, 0.0625, align63, 0.0625, align64, 0.0625)

align57
use4dct ? dct4_1 : last
use8dct ? dct8_1 : last
use32dct ? average(dct16_1, 0.5, dct32_2, 0.5) : last
use64dct ? average(dct16_1, 0.25, dct48_3, 0.25, dct32_2, 0.25, dct64_4, 0.25) : last
mergechroma(source, 1)

return(last)
}

function dfilter(clip c, int offx, int offy, float dc, float dct2, float dct3, float dct4, float dct5, float dct6, float dct7, float dct8) {
#c.PointResize(c.width,c.height,offx,offy,c.width,c.height).dctfilter(dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8,chroma=0,offx=offx,offy=offy).PointResize(c.width,c.height,-offx,-offy,c.width,c.height)
c.dctfilter(dc,dct2,dct3,dct4,dct5,dct6,dct7,dct8,chroma=0,offx=offx,offy=offy)
}


(The dctlimit stuff is something I should have done the first time I cleaned it up, didn't affect anything.)

Nonetheless, I think I'm going to look at CAFxX's filter for now, unless I have any sudden insights on how to make this more efficient.

I think the current version leans too heavily on warpsharp, it really has a rather dull and hollow look at 4x, very much like EKG - toning it down looks rather nicer imho. Still, I'm having a hard time choosing between this and Photozoom2, my other current favorite. PZ2 has much more deviation in line thickness that is very pleasant, but way too much contrast, where WD retains an enormous amount of detail, but somehow or another, those thick lines could really use some help. I only wish I knew how to get that effect.

[niiyan]

@Merlin7777
I have already downloaded a few times via IE6 and Firefox, after removing those browsers' cache.
But I couldn't extact it.
Anyway, thank you for your reply.

@foxyshadis
I downloaded a new version (DctFilter.zip 12-Feb-2007 04:46 224k).
It was successfully decompressed.
Thanks.

[*.mp4 guy]

I've been finding what settings work the best, and the new defaults should perform a lot better then the old ones, tell me if it helps with the lines. The new defaults are also a lot faster. I was having trouble with chroma upsizing, so wdresize is now just luma, while wdresizeC is luma+chroma. The thickened lines where a result of a few things (mostly dehaloing), so it should be less of an issue now, since dehaloing isn't as important as I thought it was.

Function DCTus(clip clp, float "sharpness", bool "u64dct"){
OX = m(16,Clp.Width)
OY = m(16,Clp.Height)
dx = ox - clp.width
dy = oy - clp.height
sharpness = default(sharpness, 2)
u64dct = default(u64dct, false)
clp.pointresize(Clp.Width*4,Clp.Height*4)
(dx+dy >=1) ? addborders(dx*4, dy*4, 0, 0) : last
clpcc = last
last.dctlimit(dct2=sharpness, dct3=0, dct4=0, dct5=0, use64dct=u64dct, use8dct=true)
(dx>=1) ? last.crop(dx*4, 0, 0, 0) : last
(dy>=1) ? last.crop(0, dy*4, 0, 0) : last
luma = last
#lanczosresize(clpcc,Clpcc.Width*4 ,Clpcc.Height*4)
#(dx>=1) ? last.crop(dx*4, 0, 0, 0) : last
#(dy>=1) ? last.crop(0, dy*4, 0, 0) : last
#chroma = last
#mergechroma(luma, chroma, 1)
return(luma)}

Function DCTds(clip clp,int destx, int desty){
clp
(desty*destx >= clp.width*clp.height/4) ? clp.spline36resize(m(32,(desty*2)), m(32,(desty*2))) : clp
(desty*destx >= clp.width*clp.height/4) ? dctlimit(use64dct=true, dct2=1, dct3=1, dct4=1, dct5=0, dct6=0, dct7=0, dct8=0) : clp
(desty*destx >= clp.width*clp.height/4) ? pointresize(destx, desty).mergechroma(spline36resize(clp, destx, desty), 1) : last

(desty*destx >= clp.width*clp.height/3.75) ? clp.spline36resize(m(32,(desty*1.875)), m(32,(desty*1.875))) : last
(desty*destx >= clp.width*clp.height/3.75) ? dctlimit(use64dct=true, dct2=1, dct3=1, dct4=1, dct5=1, dct6=0, dct7=0, dct8=0) : last
(desty*destx >= clp.width*clp.height/3.75) ? spline36resize(destx, desty).mergechroma(spline36resize(clp, destx, desty), 1) : last

(desty*destx >= clp.width*clp.height/3.5) ? clp.spline36resize(m(32,(desty*1.75)), m(32,(desty*1.75))) : last
(desty*destx >= clp.width*clp.height/3.5) ? dctlimit(use64dct=true, dct2=1, dct3=1, dct4=1, dct5=1, dct6=1, dct7=0, dct8=0) : last
(desty*destx >= clp.width*clp.height/3.5) ? spline36resize(destx, desty).mergechroma(spline36resize(clp, destx, desty), 1) : last

(desty*destx >= clp.width*clp.height/3.25) ? clp.spline36resize(m(32,(desty*1.625)), m(32,(desty*1.625))) : last
(desty*destx >= clp.width*clp.height/3.25) ? dctlimit(use64dct=true, dct2=1, dct3=1, dct4=1, dct5=1, dct6=1, dct7=1, dct8=0) : last
(desty*destx >= clp.width*clp.height/3.25) ? spline36resize(destx, desty).mergechroma(spline36resize(clp, destx, desty), 1) : last
return last}

function wdresize(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep", int "antihalo", int "taps", int "dehalo", float "sharpness2"){
sharpness = default(sharpness, 1.625)
u64dct = default(u64dct, true)
warp = default(warp, 0)
thresh = default(thresh, 50)
rep = default(rep, 1)
antihalo = default(antihalo, 0)
taps = default(taps, 4)
dehalo = default(dehalo, 0)
sharpness2 = default(sharpness2, 1.126)

cw = clp.width
ch = clp.height
clp
(taps <=3) ? clp.dctus(sharpness=sharpness, u64dct=u64dct) : last

(taps >=3) ? clp.pointresize(cw*2, ch*2) : last
w = last.width
h = last.height
(taps <=3) ? pointresize(w/2, h/2) : last

OX = m(16,last.Width)
OY = m(16,last.Height)
dx = ox - last.width
dy = oy - last.height
(dx+dy >=1) ? addborders(dx*4, dy*4, 0, 0) : last

clpcc = last
clpcc

(taps <=3) ? last.DeHalo_alpha(ss=1, lowsens=antihalo, highsens=antihalo, rx=2, ry=2) : last

(taps >=3) ? last.dctlimit(use64dct=u64dct, dct2=1+(sharpness/11.6), dct3=1+(sharpness/8.1), dct4=sharpness, dct5=sharpness2/8.1, dct6=sharpness2/11.6, dct7=sharpness2/16.1, dct8=sharpness2/23.1) : last

(dehalo >=1) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last



#(dx+dy >=1) ? addborders(dx*4, dy*4, 0, 0) : last

(rep>=2) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=2) ? last.DeHalo_alpha(ss=1, lowsens=antihalo, highsens=antihalo, rx=2, ry=2) : last

(rep>=3) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=3) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=4) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=4) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=5) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=5) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=6) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=6) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=7) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=7) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

(rep>=8) ? last.dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=0.5, dct5=sharpness, dct6=sharpness2/8).dctlimit(use64dct=u64dct, dct2=1, dct3=1, dct4=sharpness, dct5=sharpness2/8) : last
(dehalo >=8) ? last.DeHalo_alpha(ss=1, lowsens=round((antihalo/2)*3), highsens=round((antihalo/2)*3), rx=2, ry=2) : last

awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)
awarpsharp(depth=warp, cm=0, blurlevel=1)

(dx>=1) ? last.crop(dx*4, 0, 0, 0) : last
(dy>=1) ? last.crop(0, dy*4, 0, 0) : last
return(last)}

function wdresizec(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep", int "antihalo", int "taps", int "dehalo", float "sharpness2"){
sharpness = default(sharpness, 1.6)
u64dct = default(u64dct, true)
warp = default(warp, 0)
thresh = default(thresh, 50)
rep = default(rep, 1)
antihalo = default(antihalo, 0)
taps = default(taps, 4)
dehalo = default(dehalo, 0)
sharpness2 = default(sharpness2, 1.126)
clp
wdresize(sharpness=sharpness,u64dct=u64dct,warp=warp,thresh=thresh,rep=rep,antihalo=antihalo,taps=taps,dehalo=dehalo, sharpness2=sharpness2)
mergechroma(lanczos4resize(clp, clp.width*2, clp.height*2), 1)
return(last)}
dctds works now, but only gives good results when downsizing a mod 32 source by a factor of 2.

@CAFxX I would like to test your filter, tell me if you find out what went wrong on my system, I'll keep looking for the problem.
[edit] It looks like avsp screwed up the formatting, I'll see what I can do about it. [edit] fixed

[CAFxX]

@*.mp4 guy
Tomorrow I hope I'll have some time to try it on my second computer. I'll let you know if it worked ASAP.

BTW, those are a few other ss:
Original (http://img185.imageshack.us/img185/8560/immagineas0.png)
Spline36 (http://img185.imageshack.us/img185/4250/1462sxn5.png)
SincResize (http://img185.imageshack.us/img185/86/1462flo7.png)

and, just to remember everyone that every rose has its thorn:
Artifacts (http://img185.imageshack.us/img185/7884/immaginecq4.png)
(4x enlargment, top: SincResize(4,6), bottom:Spline36())

[*.mp4 guy]

@*.mp4 guy
Tomorrow I hope I'll have some time to try it on my second computer. I'll let you know if it worked ASAP.

BTW, those are a few other ss:
Original (http://img185.imageshack.us/img185/8560/immagineas0.png)
Spline36 (http://img185.imageshack.us/img185/4250/1462sxn5.png)
SincResize (http://img185.imageshack.us/img185/86/1462flo7.png)

and, just to remember everyone that every rose has its thorn:
Artifacts (http://img185.imageshack.us/img185/7884/immaginecq4.png)
(4x enlargment, top: SincResize(4,6), bottom:Spline36())
Is that using an fft or dft based aproach, or is that using dcts or dsts. You should get better results if you stick to powers of 2, ie 2-4-8-16, which usually compute faster and give better quality (because they are easier to comput accuracy is better). The dct and dst should greatly outperform there fourier counterparts at the same tap value.

[CAFxX]

ATM I use IPP's ippiDCTFwd_32f_C1R and ippiDCTInv_32f_C1R on blocks of size 2^Window (DCT) and (2^Window)*Factor (iDCT). They work on single precision floating point data: nevertheless all that haloing looks rather strange to me.

[CAFxX]

OK, I did a few tests and discovered that *probably* the "Unable to load plugin" problem was caused by MSVC that was inserting manifests during linking...
Now it should work. At least, it does on my second PC.
Notable changes:
- switched to IPP's internal memory allocator (ippiMalloc) for better (?) alignment
- removed multi-threading (I did one of the most stupid errors of my programming experience...)
- removed silly "GenuineIntel" check from IPP library (SIMD code should now be executed on AMD chips as well)
- CRT library now is statically linked
- removed manifest

known bugs:
- I'm investigating a bug occurring with big frame sizes

download:
http://cafxx.strayorange.com/SincResize3.7z
enjoy!

[*.mp4 guy]

It works now, its also nice and fast :). good job. The ringing has me a bit baffled though, since dctlimit doesn't have it when used with equivelent settings. Just one request.

Could you remove the restriction that factor must be greater then 1. It should be ok to use a factor of 1, and being able to do that will make it easier for me to find out where the ringing is coming from. Thanks.

[CAFxX]

There you go.

notable changes:
-factor can now be 1

download:
http://cafxx.strayorange.com/SincResize4.7z

[*.mp4 guy]

Thanks, off to test.

[*.mp4 guy]

Factor 1 doesn't look like it changes the image at all, I'm not even sure if there is any processing being done. However it looks like the intel ipp routines have lower accuracy on "high energy" frequencies then the dctfilter+dctlimit routines, it might be worth seeing if higher amounts of overlapping improve the situation, but the ideal solution would be higher accuracy routines with less roundoff error. Thanks to foxyshadis dctfilter optimisations dctlimit is fast enough (imo) to be worth using considering that it can avoid ringing almost completely.

Thanks for sharing your plugin CAFxX, It looks like there isn't an easy way to do frequency based interpolation, you either sacrifice speed, effectiveness, or artifacts, spline sacrifices effectiveness, lanczos artifacts and effectiveness both, dctlimit sacrifices speed with slight artifacts, and sincresize sacrifices artifacts and a small amount of speed =/.

At this point it looks like the next step to better performance frequency based interpolation is to write a custom aproximation of the KLT transform specifically for resizing, which is far beyond both my coding and math knowledge, I'll have to come back to the problem later when I have a better understanding of high level trigonometry and algebra (I beleive trig is heavily used in the dct) and linear matrix transformation equations.

Anyone is of course welcome to take any of my code and ideas from this thread and improve upon them themselves, though I can't speak for the other contributors.

[CAFxX]

Factor 1 doesn't look like it changes the image at all, I'm not even sure if there is any processing being done.
I noticed that as well, but the the DCT/iDCT is done nonetheless.
However it looks like the intel ipp routines have lower accuracy on "high energy" frequencies then the dctfilter+dctlimit routines, it might be worth seeing if higher amounts of overlapping improve the situation, but the ideal solution would be higher accuracy routines with less roundoff error.
In the meanwhile I've been looking for other (hopefully more precise) implementations. If I manage to find something suitable, I'll drop IPP's DCT/iDCT.
At this point it looks like the next step to better performance frequency based interpolation is to write a custom aproximation of the KLT transform specifically for resizing
One of the alternatives I'm investigating on is a template metaprogramming-based recursive transform. A solution that coupled with an arbitrary precision integer/rational/fixed-point/floating-point math library should resolve the issue.

[CAFxX]

Just a quick note to let you know that I'm making a few changes to my website, so the previous links may break in a near future.
I nevertheless created a page dedicated to SincResize that will contain the updated links:
http://cafxx.strayorange.com/SincResize
(still, new version announcments will be made here though)

[psme]

sorry, how to use dctus or wdresizec for upscale? How to specific output picture x/y size? Or the filter is not meant for such usage? Thanks in advance.

regards,

Li On

[foxyshadis]

You resize to the nearest power of 2 multiple with wdresize or wdresizec, then resize the rest of the way up or down with your favorite resizer. (Lanczos4 or Spline36, probably.) There's no way to resize in just one direction, but it wouldn't be hard to hack in.

[*.mp4 guy]

Updated wdresizec function, now supports vertical only and horizontal only resizing. Mode=1 gives vertical only resizing, mode=2 gives horizontal only, if mode is anything other then 1 or 2 normal resizing will be used.

function wdresizec(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep", int "antihalo", int "taps", int "dehalo", float "sharpness2", int "mode"){
sharpness = default(sharpness, 2)
u64dct = default(u64dct, true)
warp = default(warp, 4)
thresh = default(thresh, 50)
rep = default(rep, 1)
antihalo = default(antihalo, 0)
taps = default(taps, 4)
dehalo = default(dehalo, 0)
sharpness2 = default(sharpness2, 2)
mode = default(mode, 0)

clp

(mode == 1) ? clp.pointresize(width*2, height) : last
(mode == 2) ? clp.pointresize(width, height*2) : last

wdresize(sharpness=sharpness,u64dct=u64dct,warp=warp,thresh=thresh,rep=rep,antihalo=antihalo,taps=taps,dehalo=dehalo, sharpness2=sharpness2)
(mode == 1) ? pointresize(clp.width, clp.height*2).mergechroma(lanczos4resize(clp, clp.width, clp.height*2), 1) : Last
(mode == 2) ? pointresize(clp.width*2, clp.height).mergechroma(lanczos4resize(clp, clp.width*2, clp.height), 1) : last

(mode < 1) ? last.mergechroma(lanczos4resize(clp, clp.width*2, clp.height*2), 1) : last
(mode > 2) ? last.mergechroma(lanczos4resize(clp, clp.width*2, clp.height*2), 1) : last

return(last)}

[*.mp4 guy]

Here is a new spatial-only dotcrawl remover, utilizing dctlimit. It will only work at full d1 resolution, 720*480 for ntsc, or 720*576 for pal, it should work equally well on any video with dotcrawl, polarity framerate and original video standard should be irrelevant. If you use it on interlaced material, or prior to ivtc/deinterlacing the interlaced parameter should be set to true, else it should be false, interlaced mode is twice as slow. The sharpness parameter is self explanetory, higher values are sharper (though still dotcrawl free) but have more noise, and can cause haloing, etc. It works by removing the frequencies where dotcrawl occurs, it is not adaptive and will process the entire image equally.

Function AntiDot(clip clp, bool "interlaced", float "sharpness"){
interlaced = default(interlaced, false)
sharpness = default(sharpness, 0.5)


clp
interlaced ? clp.converttoyv12(interlaced=true).assumetff() : last
s1 = last

interlaced ? last.bob.dctlimit(use64dct=true, dct2=1+(sharpness/4), dct3=1+(sharpness/2), dct4=0, dct5=0, dct6=0, dct7=1+(sharpness/2), dct8=1+(sharpness/4)) : last.dctlimit(use64dct=true, dct2=1+(sharpness/4), dct3=1+(sharpness/2), dct4=0, dct5=0, dct6=0, dct7=1+(sharpness/2), dct8=1+(sharpness/4))
s2 = last

even = s2.selecteven.separatefields.selectodd
odd = s2.selectodd.separatefields.selecteven
interlaced ? interleave(even, odd).weave : s2

return(last)}

[edit] one of the coeficients was wrong, fixed.

[foxyshadis]

A little thing you might want to play with, inserted into wdresize between dctlimit and warpsharp, along with the function parameter:

edi = default(edi,false)
#emap=eedi2(map=1,mthresh=[<"mthresh", 0, 100, 35>],lthresh=[<"lthresh", 0, 100, 28>],vthresh=[<"vthresh", 0, 100, 28>],estr=[<"estr", 0, 8, 5>],dstr=[<"dstr", 0, 8, 5>]).gaussresize(width*2,height*2)
emap=clp.eedi2(map=1,mthresh=35,lthresh=28,vthresh=28,estr=5,dstr=5).gaussresize(width,height)
edi1=clp.eedi2.turnright.eedi2.turnleft
edi2=clp.turnright.eedi2.turnleft.eedi2
edif=mt_average(edi1,edi2).AddBorders(0,0,2,2).blackmanresize(width,height,.5,.5,-1.5,-1.5,taps=8)
edi ? mt_merge(last,edif,emap) : last

The EEDI map parameters were chosen to minimize the EDI area to only the strongest and most obvious lines. Coincidentally these are often the ones that need the most help with wd. The doubled-up EEDIs are unnecessary, just an attempt to ensure there's less artifacts. The blackman was just, well, no good reason at all, maybe lanczos or spline36 would work just as well. :p Need something to take care of the half-pixel shift though.

The addborders will cause a grey line to show up on right/bottom the edges, I just used it because it's simpler than actual extension for now.

I work with a lot of artwork, so EDI can be pretty important sometimes. On film, it's less so.

[*.mp4 guy]

I didn't know you could use eedi to make edgemaps, it looks interesting. I'll have to try it out on some of the test subjects I've been using. Depending upon how robust the edgemaps are this could be very usefull for film content aswell.

[foxyshadis]

Here's what I ended up with before my brain shut down last night:

ediclp=clp.vmtoon(sharpen=false,thinning=0,strength=35,luma_cap=203,threshold=8)
emap=ediclp.edimap(mode=edimode).gaussresize(width,height)
edi1=ediclp.eedi2.turnright.eedi2.turnleft
edi2=ediclp.turnright.eedi2.turnleft.eedi2
edif=mt_average(edi1,edi2).AddBorders(0,0,2,2).blackmanresize(width,height,.5,.5,-1.5,-1.5,taps=8) \ .vmtoon(sharpen=false,thinning=0,strength=100,luma_cap=81,threshold=6).LimitedSharpenFaster(smode=4,LMode=3,strength=20)
edi ? mt_merge(last,edif,emap) : last


function edimap(clip c, int "mode") {
mode=default(mode,2)
c
mode==1 ? eedi2(map=1,mthresh=40,lthresh=32,vthresh=28,estr=5,dstr=5) : \
mode==3 ? eedi2(map=1,mthresh=28,lthresh=22,vthresh=28,estr=3,dstr=5) : \
mode==4 ? eedi2(map=1) : \
eedi2(map=1,mthresh=35,lthresh=28,vthresh=28,estr=5,dstr=5)
}

The new function and the edimode parameter to use it came once I realized that some cartoony things need quite a bit more edi love at 4x than others. 1 is practically all dct, 4 is basically just using dctlimit as an alternative to the standard (bilinear?) interior resize for non-edge areas. I normally only use edi mode for the first 2x, then pure dct for the next. Still trying to come up with an even better way of integrating them than a simple edgemask.

The vmtoon stuff darkens thin lines similar to dctlimit, otherwise there was a pretty obvious mismatch between the two. (It won't brighten thin white lines though.) The settings just came from fiddling in AVSp and could probably use more tuning, but I have to start somewhere!

[*.mp4 guy]

I think the shift compensation is off in your script, this should compensate the eedi2 pixel shifts better, In some quick testing it aligned perfectly with a reference clip.

for standalone testing:
c = last
w = last.width
h = last.height

edi1=c.eedi2.Spline64Resize(W*1, H*2, -0.0, -0.0, W*1, H*2).turnright.eedi2.Spline64Resize(h*2, w*2, +0.5, -0.5, h*2, w*2).turnleft
edi2=c.turnright.eedi2.Spline64Resize(h*1, w*2, -0.0, -0.0, h*1, w*2).turnleft.eedi2.Spline64Resize(W*2, H*2, -0.5, -0.5, W*2, H*2)
edif=mt_average(edi1,edi2)

edif

for integration into wdresize:
edi1=clp.eedi2.Spline64Resize(cw*1, ch*2, -0.0, -0.0, cw*1, ch*2).turnright.eedi2.Spline64Resize(ch*2, cw*2, +0.5, -0.5, ch*2, cw*2).turnleft
edi2=clp.turnright.eedi2.Spline64Resize(ch*1, cw*2, -0.0, -0.0, ch*1, cw*2).turnleft.eedi2.Spline64Resize(cw*2, ch*2, -0.5, -0.5, cw*2, ch*2)

[foxyshadis]

Isn't Resize(W,H,0,0,W,H) - aka (W,H,0,0,0,0) - just a no-op, since those are all the default values?

And wouldn't the second set actually be stretching it, where the top/left margins are now offset by half a pixel, decreasing to zero at the bottom/right margins? I would expect you'd need (.5,.5,W+.5,H+.5) to shift the whole canvas half a pixel both ways - the addborders step was just a lame enabler of that since I thought you couldn't go off the edges in the crop parameters. But I freely admit I might still be ignorant.

[krieger2005]

I think this is what you searching for: http://forum.doom9.org/showthread.php?t=120759

[foxyshadis]

Oh, the second parameter is from the cropping start, not input start, if it's positive. I seeeeeeeeeeeeee. Sorry for all the confusion then.

[*.mp4 guy]

Hmm, my code was off aswell, this should fix the problem.

edi1=clp.eedi2.Spline64Resize(cw*1, ch*2, +0.5, +0.75, cw*1, ch*2).turnright.eedi2.Spline64Resize(ch*2, cw*2, +0.5, -0.5, ch*2, cw*2).turnleft
edi2=clp.turnright.eedi2.Spline64Resize(ch*1, cw*2, -0.5, +0.75, ch*1, cw*2).turnleft.eedi2.Spline64Resize(cw*2, ch*2, -0.5, -0.5, cw*2, ch*2)

This was made using trial and error, so it could also still be off, but if it is it should be less off then the other ones.

@foxyshadis
I had to search around a bit for a cartoon source to test your additions on. I'm using the first ghost in the shell movie now. EEDI2 doesn't improve most scenes, but it does help with aliasing that dctlimit misses from time to time. On particularly aliasing prone sources it would be very usefull, it also gets rid of the "pseodo aliasing" that can occur on very shallowly angled lines, which I havent been able to remove any other way.




Here is a new function, by default (mode=0, or Mode>4, mode>4 uses soft thresholding, mode=0 uses hard thresholding) it uses a mask of "areas where dct->idct will create errors" and uses dctlimit to sharpen only things outside of those areas. mode 1 returns the mask, mode 2 returns the result of dct->idct, mode 3 retuns the the sharpened results without using masking, mode 4 uses the mask to dering/dealias the image using frfun7. Thresh sets how much error is acceptable (high thresh allows processing of areas with more dct->idct error by changing the mask). Strength sets the strength of sharpening in modes 0, >4 and 3.

function dctE(clip c, int "thresh", int "strength", int "dct2", int "dct3", int "dct4", int "dct5", int "dct6", int "dct7", int "dct8", int "mode"){
thresh = default(thresh, 0)
Strength = default(strength, 3)
mode = default(mode, 0)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 2)
dct6 = default(dct6, 2)
dct7 = default(dct7, 2)
dct8 = default(dct8, 2)



s = c
s
fd1 = dctlimit(dct2=1/dct2, dct3=1/dct3, dct4=1/dct4, dct5=1/dct5, dct6=1/dct6, dct7=1/dct7, dct8=1/dct8)
fre1 = fd1.dctlimit(dct2=dct2, dct3=dct3, dct4=dct4, dct5=dct5, dct6=dct6, dct7=dct7, dct8=dct8)

m1 = yv12lutxy(fre1, s, "x y - abs "+string(thresh)+" - 256 *", u=3, v=3)
s1 = yv12lutxy(fd1, s, "y x - "+string(strength)+" * y +", u=3, v=3)

s
(mode > 4) ? maskedmerge(s1, s, dctlimit(m1)) : last
(mode < 1) ? maskedmerge(s1, s, m1) : last
(mode == 1) ? m1 : last
(mode == 2) ? fre1 : last
(mode == 3) ? s1 :last
(mode == 4) ? maskedmerge(s1, frfun7(s1, 1.5, 128), dctlimit(m1)) :last

return(last)}

The deringing/aliasing mode works very well at removing ringing on upsized anime.

[*.mp4 guy]

New version of dctE, mode 5 is better for ringing removal then mode 4, mode 6 is better for ringing+aliasing removal.
function dctE(clip c, int "thresh", int "strength", int "dct2", int "dct3", int "dct4", int "dct5", int "dct6", int "dct7", int "dct8", int "mode"){
thresh = default(thresh, 0)
Strength = default(strength, 3)
mode = default(mode, 0)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 2)
dct6 = default(dct6, 2)
dct7 = default(dct7, 2)
dct8 = default(dct8, 2)



s = c
s
fd1 = dctlimit(dct2=1/dct2, dct3=1/dct3, dct4=1/dct4, dct5=1/dct5, dct6=1/dct6, dct7=1/dct7, dct8=1/dct8)
fre1 = fd1.dctlimit(dct2=dct2, dct3=dct3, dct4=dct4, dct5=dct5, dct6=dct6, dct7=dct7, dct8=dct8)

m1 = yv12lutxy(fre1, s, "x y - abs "+string(thresh)+" - 256 *", u=3, v=3)
s1 = yv12lutxy(fd1, s, "y x - "+string(strength)+" * y +", u=3, v=3)

s
(mode > 6) ? maskedmerge(s1, s, dctlimit(m1)) : last
(mode < 1) ? maskedmerge(s1, s, m1) : last
(mode == 1) ? m1 : last
(mode == 2) ? fre1 : last
(mode == 3) ? s1 :last
(mode == 4) ? maskedmerge(s1, frfun7(s1, 1.5, 128), dctlimit(m1)) :last
(mode == 5) ? maskedmerge(s1, dctfun4b(s1, round(256/thresh), round(256/thresh)), dctlimit(m1)) :last
(mode == 6) ? maskedmerge(s1, frfun7(maskedmerge(s1, dctfun4b(s1, round(256/thresh), round(256/thresh)), dctlimit(m1)), 1.001, 128), dctlimit(m1)) :last

return(last)}

I would like some feedback on the quality of this set of paramters, right now they look to be performing better then anything else most of the time.

wdresizeC(sharpness=2.0, dehalo=0, warp=3, taps=4, u64dct=true, rep=2)
dctE(thresh=7, mode=6, dct5=8, dct6=8, dct7=8, dct8=8)

[edit] on very large upsizes I sometimes get better results with higher thresh settings on the first upsize. I changed the strength of frfun7 in mode 6 from 1.1 to 1.001 aswell, which might give better quality.

[SilaSurfer]

Sorry for bringing this thread back from slumber, but I can't find DctFilter plugin modified from foxyshadis from this post

http://forum.doom9.org/showpost.php?p=951432&postcount=10

The page is down.

If anyone has it or knows where I could find it I would be grateful. Thanks

[Dogway]

Found it a few days back dont remember where:
http://www.mediafire.com/?xjqt81fo1w3b7qd

[SilaSurfer]

:thanks: Dogway. I appreciate it.

[SilaSurfer]

First of all why I'm doubleposting because I have a question for mp4guy?

Could your function Dcte be used after downsizing for example 1080p to 720p for removal of ringing and aliasing?

[*.mp4 guy]

Maybe, I haven't used that function in a long time. Do you have a specifically problematic source, or do you just want better downscaling? Either way I need an example that displays the problem. I would be more specific, but I don't want to give you a solution that doesn't work well.

[SilaSurfer]

Sorry Mp4.guy for not being nore specific with my question and because I can't provide the examples right now which are bluray sources. I'm on the road (work) and I don't have any examples here with me on my laptop. For dowsampling I use Spline36. In my opinion it is best downscalar among avisynth resamplers but they do cause ringing at least less than Lanczos. If I sharpen a little bit before downsampling to combat that blurriness I get ringing in my output. I know that sharpening is a tradeoff between those two. I found your script on the forum under fractal sharpening thread.

Frequency limited SeeSaw

unsharp(vary=3)
s = last
sharp = SeeSaw(s,s).SeeSaw().SeeSaw().SeeSaw().SeeSaw()
diff = mt_makediff(sharp, last).spline36resize(round(width*1.5/8)*8, round(height*1.5/8)*8).dctlimit(use64dct=false, use32dct=false, use8dct=true, dc=1, dct2=0, dct3=0.25, dct4=1, dct5=1, dct6=1, dct7=1, dct8=1).spline36resize(s.width, s.height)
mt_adddiff(last, diff)

And I gave it a whirl before going on a trip and the results were awasome. Very sharp image without ringing, but too much for my taste. You see why I'm interested in your function dcte.

I would like some explenation on the ringing/aliasing removal and sharpening in your function.

Here is a new function, by default (mode=0, or Mode>4, mode>4 uses soft thresholding, mode=0 uses hard thresholding) it uses a mask of "areas where dct->idct will create errors" and uses dctlimit to sharpen only things outside of those areas (this is what caught my eye);). mode 1 returns the mask, mode 2 returns the result of dct->idct, mode 3 retuns the the sharpened results without using masking, mode 4 uses the mask to dering/dealias the image using frfun7. Thresh sets how much error is acceptable (high thresh allows processing of areas with more dct->idct error by changing the mask). Strength sets the strength of sharpening in modes 0, >4 and 3.

Thank you for your time and patience.;)

[*.mp4 guy]

DCTE is a very targeted function, its specific aim being to enhance upscales made with dctlimit. It does this by approximating a semi-reversible lowpass then performing an aproximate inverse of this
operation, and measuring the error introduced by this two step process, areas of high error are considered to have been likely problematic for the upsampler, and are either left alone, or smoothed
in some way, areas of low error are considered "safe" and are targeted for detail enhancement via a sharpening kernel derived from the earlier frequency operations or left alone.

The quality of any normal downscaling operation from a relatively intact source is very likely to be of excessively high quality compared to DCTE's intended usage case. IE it won't work unless things
are messed up.

The frequency limited seesaw script does not use DCTE, or any similar operation. In effect what is being done here, is that seesaw is used normally, then dctlimit is employed as a bandpass filter in
order to remove some of seesaw's sharpening, thereby better approximating the "fractal sharpener".

You mention that the sharpening is too strong, that is the general problem with these things, one-size-fits-all processing just doesn't work. There is always a better solution then what you are
using, and always a worse.


I would point you to some good sharpeners, but you seem to know where most of them are, so its just a matter of applying them well.

Now to removing ringing, you can't. Or more precisely its a bad idea to try for a perfectly gibbs free output, eliminating the first lobe especially never ends well. Other lobes however, can be avoided
safely, but removing them after the fact is an even worse idea then trying to eliminate the first lobe.

... and so here are two downscaling methods that ring significantly less then spline36.

function gwsmf(clip c, int w, int h)
{c
Blackmanresize(w*2, height, taps=7)
mt_convolution(horizontal=" -23 0 2912 0 -53053 0 480480 861432 480480 0 -53053 0 2912 0 -23 ", vertical=" 1 ", u=3, v=3)
mt_convolution(horizontal=" 1 -3 6 4 -38 70 -3 -222 399 -38 -1150 2584 6322 2584 -1150 -38 399 -222 -3 70 -38 4 6 -3 1 ", vertical=" 1 ", u=3, v=3)
pointresize(w, height)
Blackmanresize(width, h*2, taps=7)
mt_convolution(horizontal=" 1 ", vertical=" -23 0 2912 0 -53053 0 480480 861432 480480 0 -53053 0 2912 0 -23 ", u=3, v=3)
mt_convolution(horizontal=" 1 ", vertical=" 1 -3 6 4 -38 70 -3 -222 399 -38 -1150 2584 6322 2584 -1150 -38 399 -222 -3 70 -38 4 6 -3 1 ", u=3, v=3)
pointresize(w, h)
}

function nlld(clip c, int w, int h)
{c
Blackmanresize(w*2, height, taps=7)
NLLH4
NLLH4t
pointresize(w, height)
Blackmanresize(width, h*2, taps=7)
NLLV4
NLLV4t
pointresize(w, h)
mergechroma(c.spline36resize(w, h), 1)
}

Function NLLV4t(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurV(1*rad, 0.486)
B2 = C.BlurV(3*rad, 0.823)
B3 = C.BlurV(5*rad, 0.898)
B4= C.BlurV(7*rad, 0.927)
B5 = C.BlurV(9*rad, 0.943)
B6 = C.BlurV(11*rad, 0.954)
B7 = C.BlurV(13*rad, 0.961)

B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLH4t(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurH(1*rad, 0.486)
B2 = C.BlurH(3*rad, 0.823)
B3 = C.BlurH(5*rad, 0.898)
B4= C.BlurH(7*rad, 0.927)
B5 = C.BlurH(9*rad, 0.943)
B6 = C.BlurH(11*rad, 0.954)
B7 = C.BlurH(13*rad, 0.961)


B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLV4(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurV(1*rad, 0.486)
B2 = C.BlurV(3*rad, 0.823)
B3 = C.BlurV(5*rad, 0.898)
B4= C.BlurV(7*rad, 0.927)
B5 = C.BlurV(9*rad, 0.943)
B6 = C.BlurV(11*rad, 0.954)
B7 = C.BlurV(13*rad, 0.961)

B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 2.273 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 2.67 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 2.9 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 3.12 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 3.33 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 3.53 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLH4(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurH(1*rad, 0.486)
B2 = C.BlurH(3*rad, 0.823)
B3 = C.BlurH(5*rad, 0.898)
B4= C.BlurH(7*rad, 0.927)
B5 = C.BlurH(9*rad, 0.943)
B6 = C.BlurH(11*rad, 0.954)
B7 = C.BlurH(13*rad, 0.961)


B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 2.273 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 2.67 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 2.9 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 3.12 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 3.33 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 3.53 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function blurH(clip c, int "rad", Float "CW")
{

Rad = Default(rad, 1)
CW = Default(CW, 0.5)

Center = C
Left = C.PointResize(C.width, C.height, -rad, 0, C.width, C.height)
Right = C.PointResize(C.width, C.height, rad, 0, C.width, C.height)

Average(Center, CW/2, Left, (1-CW)/2, Right, (1-CW)/2, Center, CW/2)

Return(last)
}

Function BlurV(clip c, int "rad", Float "CW")
{

Rad = Default(rad, 1)
CW = Default(CW, 0.5)

Center = C
Down = C.PointResize(C.width, C.height, 0, -rad, C.width, C.height)
Up = C.PointResize(C.width, C.height, 0, rad, C.width, C.height)

Average(Center, CW/2, Down, (1-CW)/2, Up, (1-CW)/2, Center, CW/2)

Return(last)
}

The first is simper, but they both do essentially the same thing. Usage would look like:ffvideosource("bluray.1080p")
gwsmf(1280, 720)# or nlld(1280, 720) don't stack either function repeatedly though, they both introduce a slight luma-chroma misalignment which I am to lazy to fix atm.

[SilaSurfer]

DCTE is a very targeted function, its specific aim being to enhance upscales made with dctlimit. It does this by approximating a semi-reversible lowpass then performing an aproximate inverse of this
operation, and measuring the error introduced by this two step process, areas of high error are considered to have been likely problematic for the upsampler, and are either left alone, or smoothed
in some way, areas of low error are considered "safe" and are targeted for detail enhancement via a sharpening kernel derived from the earlier frequency operations or left alone.

The quality of any normal downscaling operation from a relatively intact source is very likely to be of excessively high quality compared to DCTE's intended usage case. IE it won't work unless things
are messed up.

Noted! :)

The frequency limited seesaw script does not use DCTE, or any similar operation. In effect what is being done here, is that seesaw is used normally, then dctlimit is employed as a bandpass filter in
order to remove some of seesaw's sharpening, thereby better approximating the "fractal sharpener".

It was an example of what dctlimit does which is also part of dcte. That is why I thought that dcte is improved version of original dctlimit.

You mention that the sharpening is too strong, that is the general problem with these things, one-size-fits-all processing just doesn't work. There is always a better solution then what you are
using, and always a worse.

I would point you to some good sharpeners, but you seem to know where most of them are, so its just a matter of applying them well.

Fully aware of that and came to a conclusion that I won't be using sharpeners when downscaling. If so maybe this one:

little detail enhancer using deen (Thank you scharfis_brain)


x1=subtract(source.deen("a2d",5,4,0),source)
result=subtract(source,x1)


5 is spatial radius
4 is luma threshold
0 is chroma threshold (you usually don't want to enhance nonexistent chroma detail, i.e. noise)

this script will only amplify faint detail/textures while leaving edges alone.

Now to removing ringing, you can't. Or more precisely its a bad idea to try for a perfectly gibbs free output, eliminating the first lobe especially never ends well. Other lobes however, can be avoided
safely, but removing them after the fact is an even worse idea then trying to eliminate the first lobe.

Tell me about it.:) Tried many things, always ended up with ringing. I won't be trying that again. Rather I'm going to switch from sharpening/ringing removal thingy, and concentrate on higher quality downscaling functions.


... and so here are two downscaling methods that ring significantly less then spline36.

The first is simper, but they both do essentially the same thing. Usage would look like:ffvideosource("bluray.1080p")
gwsmf(1280, 720)# or nlld(1280, 720) don't stack either function repeatedly though, they both introduce a slight luma-chroma misalignment which I am to lazy to fix atm.

Thank you very much Mp4guy for your time and effort. Can't wait to get home to try it out especially knowing that those two come from you. Thank you again, wish you all the best. :)

Update!

Which version of masktools2 is needed for those two functions?

[*.mp4 guy]

Which version of masktools2 is needed for those two functions?

Almost any version of masktools 2 should work fine, anything at all recent should work. nlld will probably work with any version, gwsmf might introduce a colorshift with very old versions.

[edit] I forgot to mention that nlld requires the "average" plugin.

[SilaSurfer]

Ok thanks for the info Mp3guy!

[javlak]

*.mp4 guy's functions are always top quality, so I don't think that my solution would be necessarily better, just maybe a bit lighter on resources; how about having your vid downscaled using a resizing function that doesn't produce ringing, like hermite, once and then again using splineXresize. Then you compare the pixel value differences and if the difference is small, you use the value of the splineXresize video; if it's large, you treat it as ringing and use the hermite one.
If I was confident on RPN I'd give it a go.

[Gavino]

If I was confident on RPN I'd give it a go.
You don't have to write it using RPN.
You can write the expression as infix (ie 'normal' notation) and use mt_polish() to convert it to RPN.
I'm surprised more people don't do this - it's much easier to get right that way.

[javlak]

Yes, you are right. I guess it will be slightly slower due to the conversion between infix and RPN, but I suppose this is the function:


function SpliceResize(clip v, int "w", int "h", int "threshold")
{
last = v
threshold = default (threshold, 128)
w=default(w, width(v))
h=default(h, height(v))
one=BicubicResize(v, w, h, b=0.0, c=0.0) #Hermite
two=Spline36Resize(v, w, h)
func = mt_polish("abs(x-y)>"+string(threshold)+"?y:x") #If the difference between x and y is above the threshold then use Hermite, else use Spline36; THANKS GAVINO
v=mt_lutxy(one,two,func) #This makes it happen
return v
}


I edited this one instead of replying to Gavino's post #76, so that I don't take over the thread. If I get more comments maybe I should start a new thread for this function.

[Gavino]

I guess it will be slightly slower due to the conversion between infix and RPN
Very slightly slower, yes, but only at compile-time (script loading), not run-time (frame serving).

func=mt_polish("((x+threshold)>y)?y:x") #If the difference between x and y is above the threshold then use Hermite, else use Spline36
You can't use the variable 'threshold' directly in the expression - you have to turn it into a string literal.
Shouldn't the condition be abs(x-y)>threshold anyway?

func = mt_polish("abs(x-y)>"+string(threshold)+"?y:x")

[*.mp4 guy]

The problem I foresee with this is that the artifacts introduced by ringing are not necessarily large, and the desired anti-aliasing effects are not necessarily small. Still its easy enough to implement. Basically, I don't think the metric is targeted enough to be useful.

[addendum]

Hmm, well, I just tested it, and it does suppress some first lobe ringing (though some still gets through, as it always will), but it doesn't do anything for ringing beyond that. I haven't done much more then glance at the output, but I find myself liking both hermite and spline better, it almost seems as though the output is blurrier then either hermite or spline alone, which I did not expect.

If you need less ringing then spline, blackmanresize with taps=2/3/4 is the fastest option that is preferable to hermite or box in my experience, but aliasing can be quite noticeable.

[SilaSurfer]

My vote goes for Spline36Resize, I will test your function javlak. Sorry Mp4guy for mispelling your name in my previous post.

[javlak]

Yeah, the thrshold has to be very low, my default is way off. From the tests I have conducted so far, values as low as 5 are fair game. I was pretty close to opening a new thread with a nice function with some nifty whathaveyous, but my HD crashed on me and I've lost all changes. So I'm starting over. Anyway, if you want to test my function, set the threshold to around 5-8...

[Didée]

Putting the filtering rationale in words, it goes like this:

- compare soft and sharp resizer
- soft resizer is the basis
- use the sharp resizer in those places where it is almost identical to the soft resizer


As mp4guy sayed, "I don't think the metric is targeted enough to be useful."

The mere difference tells nothing about what kind the difference is of. And some years of experience have told: if a filter needs very-very-low thresholds, then usually the concept is flawed.

[javlak]

I will agree to the very-very-low thresholds theory didee, but bear in mind that this is not a filter, it's just a function that splices two resizing algorithms together in order to avoid the disadvantages they have to offer. So really, you expect the threshold to be low; granted, one picture will be soft and the other one sharp, but on a per-pixel basis, the difference shouldn't be that large between them in the first place, hence the low threshold.

And yes, I agree with your other point; just because there's a large difference, that doesn't mean that it's necessarily an artefact. This is just an assumption and it could very well be wrong in some cases, it could be right in others etc. It's just a theory which could prove to be horrid, or maybe it won't matter to the overall perception to the eye.

But I say, let's see how badly/well it performs in practice and then bash it!

[Didée]

Okay, tried it. Conclusion with 1080p as input:

SpliceResize(960,540,6) is perceptual identical to BicubicResize(960,540,.0,.0). Needed 400% zoom, single-frame-flipping and sharp eyes to notice any differences at all.

Need to post screenshots, showing there's nothing to show? ;)

[javlak]

Heehee, as I said earlier I've updated the algorithm, I'm just writing again it now as my HD crashed and I lost it. Give me a few hours to finish it (again) and start a new thread so we stop hijacking this thread and then we open up a grave and bury it, OK? :)

Sorry about the delay, I had too many things on my hands. Here's the thread: http://forum.doom9.org/showthread.php?p=1505245

[Dogway]

For anyone using DctFilter_test.dll labeled as version 0.0.1.5 I wanted to warn that it has a bug in memory handling or something. I switched back to version 1.4 which is the one you can find in warpenterprises and the artifacts are gone, besides it's also much faster. It was really hard to track because this happens quite sporadic.

http://img193.imageshack.us/img193/7722/ffv1bug.th.png (http://img193.imageshack.us/img193/7722/ffv1bug.png)

[*.mp4 guy]

Those look very much like decoding errors to me. Anyway, the alternate version of dctfilter is specifically intended to reduce memory usage when multiple instances are used in a script, if you aren't running into stability issues due to memory overuse by dctfilter, there is no reason to use the modded version.

[Dogway]

Yes, that was my first thought as you can see how I named the pic. But I doubted it could be ffms2 decoding ffv1, moreover when I replaced it with v.0.4 and had no problems.
To be 100% sure I just passed a test loading with avisource, and the bug was still there. NPI, just a warn since I had so many headaches trying to fix this problem since long ago.

[Gser]

I wonder if it would be possible to get my hands on a 64-bit version of dctfilter.

[*.mp4 guy]

Yes, that was my first thought as you can see how I named the pic. But I doubted it could be ffms2 decoding ffv1, moreover when I replaced it with v.0.4 and had no problems.
To be 100% sure I just passed a test loading with avisource, and the bug was still there. NPI, just a warn since I had so many headaches trying to fix this problem since long ago.

I've used the moded version an incredible number of times and nothing like that ever happened. However it sounds like you did everything to isolate the problem correctly, in which case it's just an incredibly odd glitch.

[Dogway]

This was the bare minimum script I got to trigger the bug. Didn't try to take masktools2 out tho.
setmtmode(5,2)
fix=ffvideoSource("C:\fix2.avi")
ffvideoSource("C:\IVTCed.avi")
setmtmode(2,2)

# Dups blending
blend=mt_average(trim(42,-1),trim(43,-1))
blend1=mt_average(trim(44,-1),trim(45,-1))
blend2=mt_average(trim(50,-1),trim(51,-1))
blend3=mt_average(trim(52,-1),trim(53,-1))
blend4=mt_average(trim(266,-1),trim(267,-1))
blend5=mt_average(trim(269,-1),trim(270,-1))
blend6=mt_average(trim(271,-1),trim(272,-1))
blend7=mt_average(trim(273,-1),trim(274,-1))
blend8=mt_average(trim(275,-1),trim(276,-1))
blend9=mt_average(trim(277,-1),trim(278,-1))

trim(0,41)+blend+blend+blend1+blend1+trim(46,49)+blend2+blend2+blend3+blend3+trim(54,255)+fix+blend4+blend4+trim(268,-1)+blend5+blend5+blend6+blend6+blend7+blend7+blend8+blend8+blend9+blend9+trim(279,0)

Deblock_QED(quant1=20, quant2=30)