Log in

View Full Version : overlapped dct, proof of concept frequency lowpass and interpolator function


Pages : 1 [2]

foxyshadis
19th February 2007, 09:51
You resize to the nearest power of 2 multiple with wdresize or wdresizec, then resize the rest of the way up or down with your favorite resizer. (Lanczos4 or Spline36, probably.) There's no way to resize in just one direction, but it wouldn't be hard to hack in.

*.mp4 guy
19th February 2007, 14:13
Updated wdresizec function, now supports vertical only and horizontal only resizing. Mode=1 gives vertical only resizing, mode=2 gives horizontal only, if mode is anything other then 1 or 2 normal resizing will be used.

function wdresizec(clip clp, float "sharpness", bool "u64dct", int "warp", int "thresh", int "rep", int "antihalo", int "taps", int "dehalo", float "sharpness2", int "mode"){
sharpness = default(sharpness, 2)
u64dct = default(u64dct, true)
warp = default(warp, 4)
thresh = default(thresh, 50)
rep = default(rep, 1)
antihalo = default(antihalo, 0)
taps = default(taps, 4)
dehalo = default(dehalo, 0)
sharpness2 = default(sharpness2, 2)
mode = default(mode, 0)

clp

(mode == 1) ? clp.pointresize(width*2, height) : last
(mode == 2) ? clp.pointresize(width, height*2) : last

wdresize(sharpness=sharpness,u64dct=u64dct,warp=warp,thresh=thresh,rep=rep,antihalo=antihalo,taps=taps,dehalo=dehalo, sharpness2=sharpness2)
(mode == 1) ? pointresize(clp.width, clp.height*2).mergechroma(lanczos4resize(clp, clp.width, clp.height*2), 1) : Last
(mode == 2) ? pointresize(clp.width*2, clp.height).mergechroma(lanczos4resize(clp, clp.width*2, clp.height), 1) : last

(mode < 1) ? last.mergechroma(lanczos4resize(clp, clp.width*2, clp.height*2), 1) : last
(mode > 2) ? last.mergechroma(lanczos4resize(clp, clp.width*2, clp.height*2), 1) : last

return(last)}

*.mp4 guy
23rd February 2007, 22:54
Here is a new spatial-only dotcrawl remover, utilizing dctlimit. It will only work at full d1 resolution, 720*480 for ntsc, or 720*576 for pal, it should work equally well on any video with dotcrawl, polarity framerate and original video standard should be irrelevant. If you use it on interlaced material, or prior to ivtc/deinterlacing the interlaced parameter should be set to true, else it should be false, interlaced mode is twice as slow. The sharpness parameter is self explanetory, higher values are sharper (though still dotcrawl free) but have more noise, and can cause haloing, etc. It works by removing the frequencies where dotcrawl occurs, it is not adaptive and will process the entire image equally.

Function AntiDot(clip clp, bool "interlaced", float "sharpness"){
interlaced = default(interlaced, false)
sharpness = default(sharpness, 0.5)


clp
interlaced ? clp.converttoyv12(interlaced=true).assumetff() : last
s1 = last

interlaced ? last.bob.dctlimit(use64dct=true, dct2=1+(sharpness/4), dct3=1+(sharpness/2), dct4=0, dct5=0, dct6=0, dct7=1+(sharpness/2), dct8=1+(sharpness/4)) : last.dctlimit(use64dct=true, dct2=1+(sharpness/4), dct3=1+(sharpness/2), dct4=0, dct5=0, dct6=0, dct7=1+(sharpness/2), dct8=1+(sharpness/4))
s2 = last

even = s2.selecteven.separatefields.selectodd
odd = s2.selectodd.separatefields.selecteven
interlaced ? interleave(even, odd).weave : s2

return(last)}

[edit] one of the coeficients was wrong, fixed.

foxyshadis
26th February 2007, 03:30
A little thing you might want to play with, inserted into wdresize between dctlimit and warpsharp, along with the function parameter:

edi = default(edi,false)
#emap=eedi2(map=1,mthresh=[<"mthresh", 0, 100, 35>],lthresh=[<"lthresh", 0, 100, 28>],vthresh=[<"vthresh", 0, 100, 28>],estr=[<"estr", 0, 8, 5>],dstr=[<"dstr", 0, 8, 5>]).gaussresize(width*2,height*2)
emap=clp.eedi2(map=1,mthresh=35,lthresh=28,vthresh=28,estr=5,dstr=5).gaussresize(width,height)
edi1=clp.eedi2.turnright.eedi2.turnleft
edi2=clp.turnright.eedi2.turnleft.eedi2
edif=mt_average(edi1,edi2).AddBorders(0,0,2,2).blackmanresize(width,height,.5,.5,-1.5,-1.5,taps=8)
edi ? mt_merge(last,edif,emap) : last

The EEDI map parameters were chosen to minimize the EDI area to only the strongest and most obvious lines. Coincidentally these are often the ones that need the most help with wd. The doubled-up EEDIs are unnecessary, just an attempt to ensure there's less artifacts. The blackman was just, well, no good reason at all, maybe lanczos or spline36 would work just as well. :p Need something to take care of the half-pixel shift though.

The addborders will cause a grey line to show up on right/bottom the edges, I just used it because it's simpler than actual extension for now.

I work with a lot of artwork, so EDI can be pretty important sometimes. On film, it's less so.

*.mp4 guy
26th February 2007, 06:24
I didn't know you could use eedi to make edgemaps, it looks interesting. I'll have to try it out on some of the test subjects I've been using. Depending upon how robust the edgemaps are this could be very usefull for film content aswell.

foxyshadis
27th February 2007, 08:24
Here's what I ended up with before my brain shut down last night:

ediclp=clp.vmtoon(sharpen=false,thinning=0,strength=35,luma_cap=203,threshold=8)
emap=ediclp.edimap(mode=edimode).gaussresize(width,height)
edi1=ediclp.eedi2.turnright.eedi2.turnleft
edi2=ediclp.turnright.eedi2.turnleft.eedi2
edif=mt_average(edi1,edi2).AddBorders(0,0,2,2).blackmanresize(width,height,.5,.5,-1.5,-1.5,taps=8) \ .vmtoon(sharpen=false,thinning=0,strength=100,luma_cap=81,threshold=6).LimitedSharpenFaster(smode=4,LMode=3,strength=20)
edi ? mt_merge(last,edif,emap) : last


function edimap(clip c, int "mode") {
mode=default(mode,2)
c
mode==1 ? eedi2(map=1,mthresh=40,lthresh=32,vthresh=28,estr=5,dstr=5) : \
mode==3 ? eedi2(map=1,mthresh=28,lthresh=22,vthresh=28,estr=3,dstr=5) : \
mode==4 ? eedi2(map=1) : \
eedi2(map=1,mthresh=35,lthresh=28,vthresh=28,estr=5,dstr=5)
}

The new function and the edimode parameter to use it came once I realized that some cartoony things need quite a bit more edi love at 4x than others. 1 is practically all dct, 4 is basically just using dctlimit as an alternative to the standard (bilinear?) interior resize for non-edge areas. I normally only use edi mode for the first 2x, then pure dct for the next. Still trying to come up with an even better way of integrating them than a simple edgemask.

The vmtoon stuff darkens thin lines similar to dctlimit, otherwise there was a pretty obvious mismatch between the two. (It won't brighten thin white lines though.) The settings just came from fiddling in AVSp and could probably use more tuning, but I have to start somewhere!

*.mp4 guy
27th February 2007, 10:12
I think the shift compensation is off in your script, this should compensate the eedi2 pixel shifts better, In some quick testing it aligned perfectly with a reference clip.

for standalone testing:
c = last
w = last.width
h = last.height

edi1=c.eedi2.Spline64Resize(W*1, H*2, -0.0, -0.0, W*1, H*2).turnright.eedi2.Spline64Resize(h*2, w*2, +0.5, -0.5, h*2, w*2).turnleft
edi2=c.turnright.eedi2.Spline64Resize(h*1, w*2, -0.0, -0.0, h*1, w*2).turnleft.eedi2.Spline64Resize(W*2, H*2, -0.5, -0.5, W*2, H*2)
edif=mt_average(edi1,edi2)

edif

for integration into wdresize:
edi1=clp.eedi2.Spline64Resize(cw*1, ch*2, -0.0, -0.0, cw*1, ch*2).turnright.eedi2.Spline64Resize(ch*2, cw*2, +0.5, -0.5, ch*2, cw*2).turnleft
edi2=clp.turnright.eedi2.Spline64Resize(ch*1, cw*2, -0.0, -0.0, ch*1, cw*2).turnleft.eedi2.Spline64Resize(cw*2, ch*2, -0.5, -0.5, cw*2, ch*2)

foxyshadis
27th February 2007, 17:27
Isn't Resize(W,H,0,0,W,H) - aka (W,H,0,0,0,0) - just a no-op, since those are all the default values?

And wouldn't the second set actually be stretching it, where the top/left margins are now offset by half a pixel, decreasing to zero at the bottom/right margins? I would expect you'd need (.5,.5,W+.5,H+.5) to shift the whole canvas half a pixel both ways - the addborders step was just a lame enabler of that since I thought you couldn't go off the edges in the crop parameters. But I freely admit I might still be ignorant.

krieger2005
27th February 2007, 20:10
I think this is what you searching for: http://forum.doom9.org/showthread.php?t=120759

foxyshadis
27th February 2007, 23:06
Oh, the second parameter is from the cropping start, not input start, if it's positive. I seeeeeeeeeeeeee. Sorry for all the confusion then.

*.mp4 guy
3rd March 2007, 11:19
Hmm, my code was off aswell, this should fix the problem.

edi1=clp.eedi2.Spline64Resize(cw*1, ch*2, +0.5, +0.75, cw*1, ch*2).turnright.eedi2.Spline64Resize(ch*2, cw*2, +0.5, -0.5, ch*2, cw*2).turnleft
edi2=clp.turnright.eedi2.Spline64Resize(ch*1, cw*2, -0.5, +0.75, ch*1, cw*2).turnleft.eedi2.Spline64Resize(cw*2, ch*2, -0.5, -0.5, cw*2, ch*2)

This was made using trial and error, so it could also still be off, but if it is it should be less off then the other ones.

@foxyshadis
I had to search around a bit for a cartoon source to test your additions on. I'm using the first ghost in the shell movie now. EEDI2 doesn't improve most scenes, but it does help with aliasing that dctlimit misses from time to time. On particularly aliasing prone sources it would be very usefull, it also gets rid of the "pseodo aliasing" that can occur on very shallowly angled lines, which I havent been able to remove any other way.




Here is a new function, by default (mode=0, or Mode>4, mode>4 uses soft thresholding, mode=0 uses hard thresholding) it uses a mask of "areas where dct->idct will create errors" and uses dctlimit to sharpen only things outside of those areas. mode 1 returns the mask, mode 2 returns the result of dct->idct, mode 3 retuns the the sharpened results without using masking, mode 4 uses the mask to dering/dealias the image using frfun7. Thresh sets how much error is acceptable (high thresh allows processing of areas with more dct->idct error by changing the mask). Strength sets the strength of sharpening in modes 0, >4 and 3.

function dctE(clip c, int "thresh", int "strength", int "dct2", int "dct3", int "dct4", int "dct5", int "dct6", int "dct7", int "dct8", int "mode"){
thresh = default(thresh, 0)
Strength = default(strength, 3)
mode = default(mode, 0)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 2)
dct6 = default(dct6, 2)
dct7 = default(dct7, 2)
dct8 = default(dct8, 2)



s = c
s
fd1 = dctlimit(dct2=1/dct2, dct3=1/dct3, dct4=1/dct4, dct5=1/dct5, dct6=1/dct6, dct7=1/dct7, dct8=1/dct8)
fre1 = fd1.dctlimit(dct2=dct2, dct3=dct3, dct4=dct4, dct5=dct5, dct6=dct6, dct7=dct7, dct8=dct8)

m1 = yv12lutxy(fre1, s, "x y - abs "+string(thresh)+" - 256 *", u=3, v=3)
s1 = yv12lutxy(fd1, s, "y x - "+string(strength)+" * y +", u=3, v=3)

s
(mode > 4) ? maskedmerge(s1, s, dctlimit(m1)) : last
(mode < 1) ? maskedmerge(s1, s, m1) : last
(mode == 1) ? m1 : last
(mode == 2) ? fre1 : last
(mode == 3) ? s1 :last
(mode == 4) ? maskedmerge(s1, frfun7(s1, 1.5, 128), dctlimit(m1)) :last

return(last)}

The deringing/aliasing mode works very well at removing ringing on upsized anime.

*.mp4 guy
5th March 2007, 02:02
New version of dctE, mode 5 is better for ringing removal then mode 4, mode 6 is better for ringing+aliasing removal.
function dctE(clip c, int "thresh", int "strength", int "dct2", int "dct3", int "dct4", int "dct5", int "dct6", int "dct7", int "dct8", int "mode"){
thresh = default(thresh, 0)
Strength = default(strength, 3)
mode = default(mode, 0)
dct2 = default(dct2, 1)
dct3 = default(dct3, 1)
dct4 = default(dct4, 1)
dct5 = default(dct5, 2)
dct6 = default(dct6, 2)
dct7 = default(dct7, 2)
dct8 = default(dct8, 2)



s = c
s
fd1 = dctlimit(dct2=1/dct2, dct3=1/dct3, dct4=1/dct4, dct5=1/dct5, dct6=1/dct6, dct7=1/dct7, dct8=1/dct8)
fre1 = fd1.dctlimit(dct2=dct2, dct3=dct3, dct4=dct4, dct5=dct5, dct6=dct6, dct7=dct7, dct8=dct8)

m1 = yv12lutxy(fre1, s, "x y - abs "+string(thresh)+" - 256 *", u=3, v=3)
s1 = yv12lutxy(fd1, s, "y x - "+string(strength)+" * y +", u=3, v=3)

s
(mode > 6) ? maskedmerge(s1, s, dctlimit(m1)) : last
(mode < 1) ? maskedmerge(s1, s, m1) : last
(mode == 1) ? m1 : last
(mode == 2) ? fre1 : last
(mode == 3) ? s1 :last
(mode == 4) ? maskedmerge(s1, frfun7(s1, 1.5, 128), dctlimit(m1)) :last
(mode == 5) ? maskedmerge(s1, dctfun4b(s1, round(256/thresh), round(256/thresh)), dctlimit(m1)) :last
(mode == 6) ? maskedmerge(s1, frfun7(maskedmerge(s1, dctfun4b(s1, round(256/thresh), round(256/thresh)), dctlimit(m1)), 1.001, 128), dctlimit(m1)) :last

return(last)}

I would like some feedback on the quality of this set of paramters, right now they look to be performing better then anything else most of the time.

wdresizeC(sharpness=2.0, dehalo=0, warp=3, taps=4, u64dct=true, rep=2)
dctE(thresh=7, mode=6, dct5=8, dct6=8, dct7=8, dct8=8)

[edit] on very large upsizes I sometimes get better results with higher thresh settings on the first upsize. I changed the strength of frfun7 in mode 6 from 1.1 to 1.001 aswell, which might give better quality.

SilaSurfer
13th March 2011, 18:57
Sorry for bringing this thread back from slumber, but I can't find DctFilter plugin modified from foxyshadis from this post

http://forum.doom9.org/showpost.php?p=951432&postcount=10

The page is down.

If anyone has it or knows where I could find it I would be grateful. Thanks

Dogway
14th March 2011, 10:53
Found it a few days back dont remember where:
http://www.mediafire.com/?xjqt81fo1w3b7qd

SilaSurfer
15th March 2011, 12:24
:thanks: Dogway. I appreciate it.

SilaSurfer
24th May 2011, 11:38
First of all why I'm doubleposting because I have a question for mp4guy?

Could your function Dcte be used after downsizing for example 1080p to 720p for removal of ringing and aliasing?

*.mp4 guy
24th May 2011, 13:45
Maybe, I haven't used that function in a long time. Do you have a specifically problematic source, or do you just want better downscaling? Either way I need an example that displays the problem. I would be more specific, but I don't want to give you a solution that doesn't work well.

SilaSurfer
26th May 2011, 10:31
Sorry Mp4.guy for not being nore specific with my question and because I can't provide the examples right now which are bluray sources. I'm on the road (work) and I don't have any examples here with me on my laptop. For dowsampling I use Spline36. In my opinion it is best downscalar among avisynth resamplers but they do cause ringing at least less than Lanczos. If I sharpen a little bit before downsampling to combat that blurriness I get ringing in my output. I know that sharpening is a tradeoff between those two. I found your script on the forum under fractal sharpening thread.

Frequency limited SeeSaw

unsharp(vary=3)
s = last
sharp = SeeSaw(s,s).SeeSaw().SeeSaw().SeeSaw().SeeSaw()
diff = mt_makediff(sharp, last).spline36resize(round(width*1.5/8)*8, round(height*1.5/8)*8).dctlimit(use64dct=false, use32dct=false, use8dct=true, dc=1, dct2=0, dct3=0.25, dct4=1, dct5=1, dct6=1, dct7=1, dct8=1).spline36resize(s.width, s.height)
mt_adddiff(last, diff)

And I gave it a whirl before going on a trip and the results were awasome. Very sharp image without ringing, but too much for my taste. You see why I'm interested in your function dcte.

I would like some explenation on the ringing/aliasing removal and sharpening in your function.

Here is a new function, by default (mode=0, or Mode>4, mode>4 uses soft thresholding, mode=0 uses hard thresholding) it uses a mask of "areas where dct->idct will create errors" and uses dctlimit to sharpen only things outside of those areas (this is what caught my eye);). mode 1 returns the mask, mode 2 returns the result of dct->idct, mode 3 retuns the the sharpened results without using masking, mode 4 uses the mask to dering/dealias the image using frfun7. Thresh sets how much error is acceptable (high thresh allows processing of areas with more dct->idct error by changing the mask). Strength sets the strength of sharpening in modes 0, >4 and 3.

Thank you for your time and patience.;)

*.mp4 guy
27th May 2011, 16:58
DCTE is a very targeted function, its specific aim being to enhance upscales made with dctlimit. It does this by approximating a semi-reversible lowpass then performing an aproximate inverse of this
operation, and measuring the error introduced by this two step process, areas of high error are considered to have been likely problematic for the upsampler, and are either left alone, or smoothed
in some way, areas of low error are considered "safe" and are targeted for detail enhancement via a sharpening kernel derived from the earlier frequency operations or left alone.

The quality of any normal downscaling operation from a relatively intact source is very likely to be of excessively high quality compared to DCTE's intended usage case. IE it won't work unless things
are messed up.

The frequency limited seesaw script does not use DCTE, or any similar operation. In effect what is being done here, is that seesaw is used normally, then dctlimit is employed as a bandpass filter in
order to remove some of seesaw's sharpening, thereby better approximating the "fractal sharpener".

You mention that the sharpening is too strong, that is the general problem with these things, one-size-fits-all processing just doesn't work. There is always a better solution then what you are
using, and always a worse.


I would point you to some good sharpeners, but you seem to know where most of them are, so its just a matter of applying them well.

Now to removing ringing, you can't. Or more precisely its a bad idea to try for a perfectly gibbs free output, eliminating the first lobe especially never ends well. Other lobes however, can be avoided
safely, but removing them after the fact is an even worse idea then trying to eliminate the first lobe.

... and so here are two downscaling methods that ring significantly less then spline36.

function gwsmf(clip c, int w, int h)
{c
Blackmanresize(w*2, height, taps=7)
mt_convolution(horizontal=" -23 0 2912 0 -53053 0 480480 861432 480480 0 -53053 0 2912 0 -23 ", vertical=" 1 ", u=3, v=3)
mt_convolution(horizontal=" 1 -3 6 4 -38 70 -3 -222 399 -38 -1150 2584 6322 2584 -1150 -38 399 -222 -3 70 -38 4 6 -3 1 ", vertical=" 1 ", u=3, v=3)
pointresize(w, height)
Blackmanresize(width, h*2, taps=7)
mt_convolution(horizontal=" 1 ", vertical=" -23 0 2912 0 -53053 0 480480 861432 480480 0 -53053 0 2912 0 -23 ", u=3, v=3)
mt_convolution(horizontal=" 1 ", vertical=" 1 -3 6 4 -38 70 -3 -222 399 -38 -1150 2584 6322 2584 -1150 -38 399 -222 -3 70 -38 4 6 -3 1 ", u=3, v=3)
pointresize(w, h)
}

function nlld(clip c, int w, int h)
{c
Blackmanresize(w*2, height, taps=7)
NLLH4
NLLH4t
pointresize(w, height)
Blackmanresize(width, h*2, taps=7)
NLLV4
NLLV4t
pointresize(w, h)
mergechroma(c.spline36resize(w, h), 1)
}

Function NLLV4t(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurV(1*rad, 0.486)
B2 = C.BlurV(3*rad, 0.823)
B3 = C.BlurV(5*rad, 0.898)
B4= C.BlurV(7*rad, 0.927)
B5 = C.BlurV(9*rad, 0.943)
B6 = C.BlurV(11*rad, 0.954)
B7 = C.BlurV(13*rad, 0.961)

B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLH4t(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurH(1*rad, 0.486)
B2 = C.BlurH(3*rad, 0.823)
B3 = C.BlurH(5*rad, 0.898)
B4= C.BlurH(7*rad, 0.927)
B5 = C.BlurH(9*rad, 0.943)
B6 = C.BlurH(11*rad, 0.954)
B7 = C.BlurH(13*rad, 0.961)


B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " x 128 - y 128 - y 128 - abs x 128 - abs / 1 > 0.5 2.71828 0.5 -1 * y 128 - abs x 128 - abs / 0.0500001 * * ^ * 1 0.5 y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * y 128 - abs x 128 - abs / 0.0500001 * 3 ^ 3 / y 128 - abs x 128 - abs / 0.0500001 * 5 ^ 3 5 * / y 128 - abs x 128 - abs / 0.0500001 * 7 ^ 3 5 * 7 * / + + + * + / * 1 ? * + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLV4(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurV(1*rad, 0.486)
B2 = C.BlurV(3*rad, 0.823)
B3 = C.BlurV(5*rad, 0.898)
B4= C.BlurV(7*rad, 0.927)
B5 = C.BlurV(9*rad, 0.943)
B6 = C.BlurV(11*rad, 0.954)
B7 = C.BlurV(13*rad, 0.961)

B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 2.273 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 2.67 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 2.9 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 3.12 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 3.33 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 3.53 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function NLLH4(Clip C, int "rad")
{


Rad = Default(Rad, 1)

B1 = C.BlurH(1*rad, 0.486)
B2 = C.BlurH(3*rad, 0.823)
B3 = C.BlurH(5*rad, 0.898)
B4= C.BlurH(7*rad, 0.927)
B5 = C.BlurH(9*rad, 0.943)
B6 = C.BlurH(11*rad, 0.954)
B7 = C.BlurH(13*rad, 0.961)


B1_D = Mt_Makediff(B1, C, u=1, v=1)
B2_D = Mt_MakeDiff(C, B2, u=1, v=1)
B3_D = Mt_MakeDiff(B3, C, u=1, v=1)
B4_D = Mt_MakeDiff(C, B4, u=1, v=1)
B5_D = Mt_Makediff(B5, C, u=1, v=1)
B6_D = Mt_MakeDiff(C, B6, u=1, v=1)
B7_D = Mt_MakeDiff(B7, C, u=1, v=1)

B2_DT = Mt_LutXY(B1_D, B2_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 2.273 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B3_DT = Mt_LutXY(B2_DT, B3_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 2.67 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B4_DT = Mt_LutXY(B3_DT, B4_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 2.9 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B5_DT = Mt_LutXY(B4_DT, B5_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.273 * X 128 - abs - 0 > Y 128 - abs 3.12 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B6_DT = Mt_LutXY(B5_DT, B6_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.67 * X 128 - abs - 0 > Y 128 - abs 3.33 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)
B7_DT = Mt_LutXY(B6_DT, B7_D, " X 128 - Y 128 - X 128 - abs Y 128 - abs * 1 + * X 128 - abs Y 128 - abs * Y 128 - abs 2.9 * X 128 - abs - 0 > Y 128 - abs 3.53 * X 128 - abs - 0 ? X 128 - abs 0 > X 128 - abs -1 X 128 - abs 0 > X 128 - 1 ? / ^ 1 ? / 1 * + 1 + / + 128 + ", u=1, v=1)

Mt_AddDiff(B7_DT, C)

#Mt_AddDiff(B1_D, B2_D)
#Mt_AddDiff(B3_D, last)
#Mt_AddDiff(B4_D, last)
#Mt_AddDiff(B5_D, last)
#Mt_AddDiff(B6_D, last)
#Mt_AddDiff(B7_D, last)
#Mt_AddDiff(C, last)


Return(last)
}

Function blurH(clip c, int "rad", Float "CW")
{

Rad = Default(rad, 1)
CW = Default(CW, 0.5)

Center = C
Left = C.PointResize(C.width, C.height, -rad, 0, C.width, C.height)
Right = C.PointResize(C.width, C.height, rad, 0, C.width, C.height)

Average(Center, CW/2, Left, (1-CW)/2, Right, (1-CW)/2, Center, CW/2)

Return(last)
}

Function BlurV(clip c, int "rad", Float "CW")
{

Rad = Default(rad, 1)
CW = Default(CW, 0.5)

Center = C
Down = C.PointResize(C.width, C.height, 0, -rad, C.width, C.height)
Up = C.PointResize(C.width, C.height, 0, rad, C.width, C.height)

Average(Center, CW/2, Down, (1-CW)/2, Up, (1-CW)/2, Center, CW/2)

Return(last)
}

The first is simper, but they both do essentially the same thing. Usage would look like:ffvideosource("bluray.1080p")
gwsmf(1280, 720)# or nlld(1280, 720) don't stack either function repeatedly though, they both introduce a slight luma-chroma misalignment which I am to lazy to fix atm.

SilaSurfer
28th May 2011, 15:24
DCTE is a very targeted function, its specific aim being to enhance upscales made with dctlimit. It does this by approximating a semi-reversible lowpass then performing an aproximate inverse of this
operation, and measuring the error introduced by this two step process, areas of high error are considered to have been likely problematic for the upsampler, and are either left alone, or smoothed
in some way, areas of low error are considered "safe" and are targeted for detail enhancement via a sharpening kernel derived from the earlier frequency operations or left alone.

The quality of any normal downscaling operation from a relatively intact source is very likely to be of excessively high quality compared to DCTE's intended usage case. IE it won't work unless things
are messed up.

Noted! :)

The frequency limited seesaw script does not use DCTE, or any similar operation. In effect what is being done here, is that seesaw is used normally, then dctlimit is employed as a bandpass filter in
order to remove some of seesaw's sharpening, thereby better approximating the "fractal sharpener".

It was an example of what dctlimit does which is also part of dcte. That is why I thought that dcte is improved version of original dctlimit.

You mention that the sharpening is too strong, that is the general problem with these things, one-size-fits-all processing just doesn't work. There is always a better solution then what you are
using, and always a worse.

I would point you to some good sharpeners, but you seem to know where most of them are, so its just a matter of applying them well.

Fully aware of that and came to a conclusion that I won't be using sharpeners when downscaling. If so maybe this one:

little detail enhancer using deen (Thank you scharfis_brain)


x1=subtract(source.deen("a2d",5,4,0),source)
result=subtract(source,x1)


5 is spatial radius
4 is luma threshold
0 is chroma threshold (you usually don't want to enhance nonexistent chroma detail, i.e. noise)

this script will only amplify faint detail/textures while leaving edges alone.

Now to removing ringing, you can't. Or more precisely its a bad idea to try for a perfectly gibbs free output, eliminating the first lobe especially never ends well. Other lobes however, can be avoided
safely, but removing them after the fact is an even worse idea then trying to eliminate the first lobe.

Tell me about it.:) Tried many things, always ended up with ringing. I won't be trying that again. Rather I'm going to switch from sharpening/ringing removal thingy, and concentrate on higher quality downscaling functions.


... and so here are two downscaling methods that ring significantly less then spline36.

The first is simper, but they both do essentially the same thing. Usage would look like:ffvideosource("bluray.1080p")
gwsmf(1280, 720)# or nlld(1280, 720) don't stack either function repeatedly though, they both introduce a slight luma-chroma misalignment which I am to lazy to fix atm.

Thank you very much Mp4guy for your time and effort. Can't wait to get home to try it out especially knowing that those two come from you. Thank you again, wish you all the best. :)

Update!

Which version of masktools2 is needed for those two functions?

*.mp4 guy
28th May 2011, 21:44
Which version of masktools2 is needed for those two functions?

Almost any version of masktools 2 should work fine, anything at all recent should work. nlld will probably work with any version, gwsmf might introduce a colorshift with very old versions.

[edit] I forgot to mention that nlld requires the "average" plugin.

SilaSurfer
29th May 2011, 14:52
Ok thanks for the info Mp3guy!

javlak
31st May 2011, 20:29
*.mp4 guy's functions are always top quality, so I don't think that my solution would be necessarily better, just maybe a bit lighter on resources; how about having your vid downscaled using a resizing function that doesn't produce ringing, like hermite, once and then again using splineXresize. Then you compare the pixel value differences and if the difference is small, you use the value of the splineXresize video; if it's large, you treat it as ringing and use the hermite one.
If I was confident on RPN I'd give it a go.

Gavino
31st May 2011, 21:18
If I was confident on RPN I'd give it a go.
You don't have to write it using RPN.
You can write the expression as infix (ie 'normal' notation) and use mt_polish() to convert it to RPN.
I'm surprised more people don't do this - it's much easier to get right that way.

javlak
31st May 2011, 22:51
Yes, you are right. I guess it will be slightly slower due to the conversion between infix and RPN, but I suppose this is the function:


function SpliceResize(clip v, int "w", int "h", int "threshold")
{
last = v
threshold = default (threshold, 128)
w=default(w, width(v))
h=default(h, height(v))
one=BicubicResize(v, w, h, b=0.0, c=0.0) #Hermite
two=Spline36Resize(v, w, h)
func = mt_polish("abs(x-y)>"+string(threshold)+"?y:x") #If the difference between x and y is above the threshold then use Hermite, else use Spline36; THANKS GAVINO
v=mt_lutxy(one,two,func) #This makes it happen
return v
}


I edited this one instead of replying to Gavino's post #76, so that I don't take over the thread. If I get more comments maybe I should start a new thread for this function.

Gavino
31st May 2011, 23:14
I guess it will be slightly slower due to the conversion between infix and RPN
Very slightly slower, yes, but only at compile-time (script loading), not run-time (frame serving).

func=mt_polish("((x+threshold)>y)?y:x") #If the difference between x and y is above the threshold then use Hermite, else use Spline36
You can't use the variable 'threshold' directly in the expression - you have to turn it into a string literal.
Shouldn't the condition be abs(x-y)>threshold anyway?

func = mt_polish("abs(x-y)>"+string(threshold)+"?y:x")

*.mp4 guy
1st June 2011, 01:23
The problem I foresee with this is that the artifacts introduced by ringing are not necessarily large, and the desired anti-aliasing effects are not necessarily small. Still its easy enough to implement. Basically, I don't think the metric is targeted enough to be useful.

[addendum]

Hmm, well, I just tested it, and it does suppress some first lobe ringing (though some still gets through, as it always will), but it doesn't do anything for ringing beyond that. I haven't done much more then glance at the output, but I find myself liking both hermite and spline better, it almost seems as though the output is blurrier then either hermite or spline alone, which I did not expect.

If you need less ringing then spline, blackmanresize with taps=2/3/4 is the fastest option that is preferable to hermite or box in my experience, but aliasing can be quite noticeable.

SilaSurfer
2nd June 2011, 15:52
My vote goes for Spline36Resize, I will test your function javlak. Sorry Mp4guy for mispelling your name in my previous post.

javlak
2nd June 2011, 17:21
Yeah, the thrshold has to be very low, my default is way off. From the tests I have conducted so far, values as low as 5 are fair game. I was pretty close to opening a new thread with a nice function with some nifty whathaveyous, but my HD crashed on me and I've lost all changes. So I'm starting over. Anyway, if you want to test my function, set the threshold to around 5-8...

Didée
2nd June 2011, 17:45
Putting the filtering rationale in words, it goes like this:

- compare soft and sharp resizer
- soft resizer is the basis
- use the sharp resizer in those places where it is almost identical to the soft resizer


As mp4guy sayed, "I don't think the metric is targeted enough to be useful."

The mere difference tells nothing about what kind the difference is of. And some years of experience have told: if a filter needs very-very-low thresholds, then usually the concept is flawed.

javlak
2nd June 2011, 18:30
I will agree to the very-very-low thresholds theory didee, but bear in mind that this is not a filter, it's just a function that splices two resizing algorithms together in order to avoid the disadvantages they have to offer. So really, you expect the threshold to be low; granted, one picture will be soft and the other one sharp, but on a per-pixel basis, the difference shouldn't be that large between them in the first place, hence the low threshold.

And yes, I agree with your other point; just because there's a large difference, that doesn't mean that it's necessarily an artefact. This is just an assumption and it could very well be wrong in some cases, it could be right in others etc. It's just a theory which could prove to be horrid, or maybe it won't matter to the overall perception to the eye.

But I say, let's see how badly/well it performs in practice and then bash it!

Didée
2nd June 2011, 19:35
Okay, tried it. Conclusion with 1080p as input:

SpliceResize(960,540,6) is perceptual identical to BicubicResize(960,540,.0,.0). Needed 400% zoom, single-frame-flipping and sharp eyes to notice any differences at all.

Need to post screenshots, showing there's nothing to show? ;)

javlak
2nd June 2011, 20:16
Heehee, as I said earlier I've updated the algorithm, I'm just writing again it now as my HD crashed and I lost it. Give me a few hours to finish it (again) and start a new thread so we stop hijacking this thread and then we open up a grave and bury it, OK? :)

Sorry about the delay, I had too many things on my hands. Here's the thread: http://forum.doom9.org/showthread.php?p=1505245

Dogway
3rd October 2011, 06:11
For anyone using DctFilter_test.dll labeled as version 0.0.1.5 I wanted to warn that it has a bug in memory handling or something. I switched back to version 1.4 which is the one you can find in warpenterprises and the artifacts are gone, besides it's also much faster. It was really hard to track because this happens quite sporadic.

http://img193.imageshack.us/img193/7722/ffv1bug.th.png (http://img193.imageshack.us/img193/7722/ffv1bug.png)

*.mp4 guy
5th October 2011, 10:20
Those look very much like decoding errors to me. Anyway, the alternate version of dctfilter is specifically intended to reduce memory usage when multiple instances are used in a script, if you aren't running into stability issues due to memory overuse by dctfilter, there is no reason to use the modded version.

Dogway
5th October 2011, 12:22
Yes, that was my first thought as you can see how I named the pic. But I doubted it could be ffms2 decoding ffv1, moreover when I replaced it with v.0.4 and had no problems.
To be 100% sure I just passed a test loading with avisource, and the bug was still there. NPI, just a warn since I had so many headaches trying to fix this problem since long ago.

Gser
5th October 2011, 12:38
I wonder if it would be possible to get my hands on a 64-bit version of dctfilter.

*.mp4 guy
6th October 2011, 20:19
Yes, that was my first thought as you can see how I named the pic. But I doubted it could be ffms2 decoding ffv1, moreover when I replaced it with v.0.4 and had no problems.
To be 100% sure I just passed a test loading with avisource, and the bug was still there. NPI, just a warn since I had so many headaches trying to fix this problem since long ago.

I've used the moded version an incredible number of times and nothing like that ever happened. However it sounds like you did everything to isolate the problem correctly, in which case it's just an incredibly odd glitch.

Dogway
6th October 2011, 20:48
This was the bare minimum script I got to trigger the bug. Didn't try to take masktools2 out tho.
setmtmode(5,2)
fix=ffvideoSource("C:\fix2.avi")
ffvideoSource("C:\IVTCed.avi")
setmtmode(2,2)

# Dups blending
blend=mt_average(trim(42,-1),trim(43,-1))
blend1=mt_average(trim(44,-1),trim(45,-1))
blend2=mt_average(trim(50,-1),trim(51,-1))
blend3=mt_average(trim(52,-1),trim(53,-1))
blend4=mt_average(trim(266,-1),trim(267,-1))
blend5=mt_average(trim(269,-1),trim(270,-1))
blend6=mt_average(trim(271,-1),trim(272,-1))
blend7=mt_average(trim(273,-1),trim(274,-1))
blend8=mt_average(trim(275,-1),trim(276,-1))
blend9=mt_average(trim(277,-1),trim(278,-1))

trim(0,41)+blend+blend+blend1+blend1+trim(46,49)+blend2+blend2+blend3+blend3+trim(54,255)+fix+blend4+blend4+trim(268,-1)+blend5+blend5+blend6+blend6+blend7+blend7+blend8+blend8+blend9+blend9+trim(279,0)

Deblock_QED(quant1=20, quant2=30)