Port of NNEDI under new v2.6 AVS API

[Reel.Deel]

Though does that mean when they finally get around to fixing the internal resizers that nnedi3 is going to be wrong again?

Not necessarily, a chroma placement parameter can be added, "MPEG1" would have to be the default in order to keep backward compatibly with the current behavior.

[Desbreko]

This fix doesn't affect santiag since it uses nnedi3 directly with dh=true rather than nnedi3_rpow2.

That said, santiag does cause chroma shift because it uses AviSynth's resizers. I think it also doesn't correct the vertical chroma shift caused by nnedi3 doubling YV12.

And yes, if/when AviSynth's resizers get fixed, nnedi3_rpow2 will need to be updated or else it will be wrong again, at least by default. With the current workaround, you'd have to set mpeg2=false when the chroma placement actually is MPEG2.

[jpsdr]

Not necessarily, a chroma placement parameter can be added, "MPEG1" would have to be the default in order to keep backward compatibly with the current behavior.

You're right indeed. I'll change that.

[jpsdr]

New version, with default value of mpeg2 parameter to false, to keep previous behavior (but don't put back chroma shift issue... ;) )

[Reel.Deel]

You're right indeed. I'll change that.

I was hypothetically speaking about AviSynth's internal resizers if the chroma placement issue ever gets fixed.

Regardless, thanks for the updated version!

---

Anyone have any toughs on this?

Since we're on the subject of speed and turning...

Currently nnedi3_rpow2() does this internally:
nnedi3()
turnright()
nnedi3()
turnleft()

It should in theory be slightly faster if rearranged to:
turnright()
nnedi3()
turnleft()
nnedi3()

I didn't bother to test it myself but you'll be turning half the number of pixels so maybe it'll be a noticable improvement. Or maybe it'll look a lot different... who knows...

[jpsdr]

Out of curiosity, i've wanted to test what's explained here (http://forum.doom9.org/showthread.php?t=170029), with the recap here (http://forum.doom9.org/showpost.php?p=1705237&postcount=34) and it's what i've used in the NNEDI version.

My script test is :

function ResizeUnderTest1(clip clp, int rf, int SizeX, int SizeY, bool chroma){
return nnedi3_rpow2(clp,rf,cshift="spline36resize",fwidth=SizeX,fheight=SizeY,mpeg2=chroma)
}

function ResizeUnderTest4(clip clp, int rf, int SizeX, int SizeY, bool chroma){
return nnedi3_rpow2(clp,rf,cshift="spline36resize",mpeg2=chroma)
}


function ResizeTest(clip clp, int rf, int SizeX, int SizeY, int p, int n, bool chroma){
Assert (n>=0)
return (n==1)
\ ? (p==1) ? ResizeUnderTest1(clp,rf,SizeX,SizeY,chroma)
\ : (p==2) ? ResizeUnderTest2(clp,rf,SizeX,SizeY,chroma)
\ : (p==3) ? ResizeUnderTest3(clp,rf,SizeX,SizeY,chroma)
\ : (p==4) ? ResizeUnderTest4(clp,rf,SizeX,SizeY,chroma)
\ : (p==5) ? ResizeUnderTest5(clp,rf,SizeX,SizeY,chroma)
\ : (p==6) ? ResizeUnderTest6(clp,rf,SizeX,SizeY,chroma)
\ : (p==7) ? ResizeUnderTest7(clp,rf,SizeX,SizeY,chroma) : ResizeUnderTest1(clp,rf,SizeX,SizeY,chroma)
\ : (p==1) ? ResizeUnderTest1(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==2) ? ResizeUnderTest2(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==3) ? ResizeUnderTest3(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==4) ? ResizeUnderTest4(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==5) ? ResizeUnderTest5(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==6) ? ResizeUnderTest6(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==7) ? ResizeUnderTest7(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma) : ResizeUnderTest1(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
}

r = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$FF0000).ConvertToYV12(ChromaOutPlacement="MPEG2")
g = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$00FF00).ConvertToYV12(ChromaOutPlacement="MPEG2")

stackhorizontal(r,g)
o=stackvertical(last,last.fliphorizontal())
X0=Width(o)
Y0=Height(o)

a=ResizeTest(o,16,500,500,4,2,true)
X1=Width(a)
Y1=Height(a)

shift=0.25*(1.0-(Float(X0)/Float(X1)))
Y = ConvertToY8(o).Spline36Resize(X1, Y1)
U = UToY8(o).Spline36Resize(X1/2,Y1/2, src_left=shift)
V = VToY8(o).Spline36Resize(X1/2,Y1/2, src_left=shift)
b = YToUV(U, V, Y)

c=Spline36Resize(o,X1,Y1)

video=interleave(a,b,c)


# Change final resolution to be watchable under VDub

#shift=0.25*(1.0-(Float(X1)/512.0))
shift=0
Y = ConvertToY8(video).Spline36Resize(512, 512)
U = UToY8(video).Spline36Resize(256,256, src_left=shift)
V = VToY8(video).Spline36Resize(256,256, src_left=shift)
YToUV(U, V, Y)


I've played a lot with it, but result is always the same : If you apply the formula of the chroma shift resize, it end up bads... :(
So, i begin to have big second thoughts about this... :confused:

Unless i'm doing something wrong, i'll welcome any feeback and even more feedback about thouse who apparently master the subject...

My purpose was of course to check the rightness of this on cases shift may produce several pixels shift.

[jpsdr]

I was hypothetically speaking about AviSynth's internal resizers if the chroma placement issue ever gets fixed.


Ah... I misunderstood your comment.

Personnaly, i would rather stay with my first choice :
Make mpeg2=true default value, and mpeg2=false affect only 4:2:0 and not 4:2:2, because 4:2:2 is always MPEG-2 aligned.

But, what the other users think, want, prefer ?

[real.finder]

Ah... I misunderstood your comment.

Personnaly, i would rather stay with my first choice :
Make mpeg2=true default value, and mpeg2=false affect only 4:2:0 and not 4:2:2, because 4:2:2 is always MPEG-2 aligned.

But, what the other users think, want, prefer ?

I think mpeg2 should be the default as dither_resize and others, and peoples encode yv12 mpeg2 almost all the time, not mpeg1 and jpeg

[Desbreko]

I also think it would be better to make mpeg2=true default for the time being and then just remove the workaround if AviSynth's resizers get fixed. That way, the majority of users will get correct behavior without having to set any new parameters since MPEG2 chroma siting is much more common, and it won't break existing scripts when it's removed later.

[jpsdr]

Ok, for now i'll add another parameter to anable or not the workaround, with a default value of true, and put back mpeg2=true the default.

[jpsdr]

I've used this script test :

function ResizeUnderTest1(clip clp, int rf, int SizeX, int SizeY, bool chroma){
return nnedi3_rpow2(clp,rf,cshift="spline36resize",fwidth=SizeX,fheight=SizeY,mpeg2=chroma)
}

function ResizeUnderTest4(clip clp, int rf, int SizeX, int SizeY, bool chroma){
return nnedi3_rpow2(clp,rf,cshift="spline36resize",mpeg2=chroma)
}


function ResizeTest(clip clp, int rf, int SizeX, int SizeY, int p, int n, bool chroma){
Assert (n>=0)
return (n==1)
\ ? (p==1) ? ResizeUnderTest1(clp,rf,SizeX,SizeY,chroma)
\ : (p==2) ? ResizeUnderTest2(clp,rf,SizeX,SizeY,chroma)
\ : (p==3) ? ResizeUnderTest3(clp,rf,SizeX,SizeY,chroma)
\ : (p==4) ? ResizeUnderTest4(clp,rf,SizeX,SizeY,chroma)
\ : (p==5) ? ResizeUnderTest5(clp,rf,SizeX,SizeY,chroma)
\ : (p==6) ? ResizeUnderTest6(clp,rf,SizeX,SizeY,chroma)
\ : (p==7) ? ResizeUnderTest7(clp,rf,SizeX,SizeY,chroma) : ResizeUnderTest1(clp,rf,SizeX,SizeY,chroma)
\ : (p==1) ? ResizeUnderTest1(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==2) ? ResizeUnderTest2(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==3) ? ResizeUnderTest3(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==4) ? ResizeUnderTest4(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==5) ? ResizeUnderTest5(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==6) ? ResizeUnderTest6(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
\ : (p==7) ? ResizeUnderTest7(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma) : ResizeUnderTest1(ResizeTest(clp,rf,SizeX,SizeY,p,n-1,chroma),rf,SizeX,SizeY,chroma)
}

r = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$FF0000).ConvertToYV12(ChromaOutPlacement="MPEG2")
g = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$00FF00).ConvertToYV12(ChromaOutPlacement="MPEG2")

stackhorizontal(r,g)
o=stackvertical(last,last.fliphorizontal())
X0=Width(o)
Y0=Height(o)

a=ResizeTest(o,32,500,500,4,1,true)
X1=Width(a)
Y1=Height(a)

shift=0.25*(1.0-(Float(X0)/Float(X1)))
Y = ConvertToY8(o).Spline36Resize(X1, Y1)
U = UToY8(o).Spline36Resize(X1/2,Y1/2, src_left=shift)
V = VToY8(o).Spline36Resize(X1/2,Y1/2, src_left=shift)
b = YToUV(U, V, Y)

c=Spline36Resize(o,X1,Y1)

d=Resize8(o,X1,Y1,kernel="Spline36Resize",kernel_c="Spline36Resize")

interleave(a,b,c,d)


Results still show that compensate the chroma according the resize don't produce good result.
Or again, i'm doing something wrong :confused:

BTW, i'm very surprised by the Resize8 result !
Never tested before, i should have....

[Desbreko]

.14 and .15's horizontal center shift correction for YV12 seems to be using a constant -0.5 luma pixel shift instead of increasing with rfactor like it should.

Edit: Looking at the source code history, I think this bug might've actually been introduced in .13, but I don't have the dll anymore to test it and only the latest release is on GitHub.

[real.finder]

I was hypothetically speaking about AviSynth's internal resizers if the chroma placement issue ever gets fixed.

Regardless, thanks for the updated version!

---

Anyone have any toughs on this?

Since we're on the subject of speed and turning...

Currently nnedi3_rpow2() does this internally:
nnedi3(dh=true)
turnright()
nnedi3(dh=true)
turnleft()

It should in theory be slightly faster if rearranged to:
turnright()
nnedi3(dh=true)
turnleft()
nnedi3(dh=true)

I didn't bother to test it myself but you'll be turning half the number of pixels so maybe it'll be a noticable improvement. Or maybe it'll look a lot different... who knows...



yes, it faster, Especially in long time encoded

[jpsdr]

.14 and .15's horizontal center shift correction for YV12 seems to be using a constant -0.5 luma pixel shift instead of increasing with rfactor like it should.


Chroma resize ajustment compensate.
It's not a bug, it is how it's supposed to be, even if i begin to wonder.

Read my post just before yours, #162...

About Turning, i'll not try changing order, it will probably mess again with the chroma shift, and if you look at the code, you'll see field parameter not allways the same, and i think it's intended to be done in the order it's done, things are not allways swapables, and i think here it's not. So, i'll not touch this.

[Desbreko]

No, it's not how it's supposed to be. It breaks the center shift correction not just for the chroma channels but for the luma as well.

g = BlankClip(width=8, height=8, pixel_type="YV12", color_yuv=$FF0000).KillAudio()
b = BlankClip(width=8, height=8, pixel_type="YV12", color_yuv=$00FF00).KillAudio()
StackVertical(StackHorizontal(g,b,g,b),StackHorizontal(b,g,b,g),StackHorizontal(g,b,g,b),StackHorizontal(b,g,b,g))

w = Width()
h = Height()

rfac = 16

r16 = Dither_convert_8_to_16().Dither_resize16(w*rfac,h*rfac).DitherPost(mode=-1)
rpow2cs = nnedi3_rpow2(rfac, cshift="Spline36Resize")
Interleave(r16,rpow2cs)
ConvertToY8()# Get rid of the chroma to avoid any possible chroma shift differences while comparing

[jpsdr]

Ok, i think i see what you mean, i thought you were talking of another thing.

[jpsdr]

New version, center & chroma should be fine this time. Add a parameter to disable the resize chroma shift adjustment.

[jpsdr]

Since we're on the subject of speed and turning...

Currently nnedi3_rpow2() does this internally:
nnedi3()
turnright()
nnedi3()
turnleft()

It should in theory be slightly faster if rearranged to:
turnright()
nnedi3()
turnleft()
nnedi3()


Finaly, i've been able to implement this properly, tested, it works, so new version in a few hours.

[ryrynz]

is it likely that nnedi3 can or would extend beyond 256 neurons?

[jpsdr]

You're talking here of something related to the core of the programm, i absolutely don't touch at that, so, it will stay the way it is.

[jpsdr]

New version up, as usual, everything is on the first post.

[Groucho2004]

11/08/2015 v0.9.4.17
+ Change the order between turnl/r and nnedi3 calls in nnedi3_rpow2 to optimize speed.
I tried this as well a few days ago, following Myrsloik's suggestion. The result was that the output is different from the original.

I just ran a test with your v0.9.4.16 vs. v0.9.4.17, the problem is the same.
Script for both versions:
colorbars(width = 320, height = 240, pixel_type = "yv12").killaudio().assumefps(50, 1).trim(0, 19)
nnedi3_rpow2(2)

Then loaded the script in VDub and created a lossless AVI (UTVideo) for each nnedi3 version.

After that, compare the two versions with this script:
a = avisource("test_pattern1.avi")
b = avisource("test_pattern2.avi")
subtract(a, b).levels(127, 1, 129, 0, 255)


Source frame:
http://s29.postimg.org/obhm0x0tj/test_src.png

Result of the diff script:
http://s12.postimg.org/nsvsvo0l9/test_diff.png

[jpsdr]

The fact that output is different doesn't realy surprise me, as we have somehow a non-linear and a non bijective transformation, and honestly i didn't expect an identical pixel result.
It doesn't mean that the result is incorrect or bad. It's different, yes, but both results are equaly good. And if one is faster than the other, he's not better, but only more advantageous.

Out of curiosity, i'm more interest to see an histogram of the subtract instead of just the Level thing.

[Desbreko]

I did a bit of testing with both methods, and while the output is different, I wouldn't call it better or worse from what I saw. For as many places where it looked worse, there were also places it looked better.

[Groucho2004]

The fact that output is different doesn't realy surprise me, as we have somehow a non-linear and a non bijective transformation, and honestly i didn't expect an identical pixel result.
It doesn't mean that the result is incorrect or bad. It's different, yes, but both results are equaly good. And if one is faster than the other, he's not better, but only more advantageous.
My point is simply that people should be aware that the output is different to the original (i.e. Tritical's 0.9.4).

[jpsdr]

Having corrected the chroma shift bug, only with that output is different from the original. Add the chroma resize adjustment for MPEG-2 subsampling, you add another differences.

All resizer work by resizing one side, and after the other. Whatever you chose to do first is irrelevant, there is no right order.
Only doing that :

Spline36Resize(video,Width(video),SizeY).Spline36Resize(SizeX,SizeY)

or

Spline36Resize(video,SizeX,Height(video)).Spline36Resize(SizeX,SizeY)

produce different results, and Spline36Resize(SizeX,SizeY) is one of the two, don't remember wicho one, but it could have been any of the two.

My point is :
Eventualy notify that because of the chroma shift bug corrected, and the MPEG-2 subsampling taking care of, output is different, eventualy.

But talking about the order the resize is done, producing slighty different result, absolutely not. Because either you have video/image processing knowledge, and in that case you know that's it's irrelevant and result is different but as good as the other, either you don't have knowledge, and in that case, people can be "frigthen" for nothing.

About the MPEG-2 thing, until now all the tests i've done showed that formula produce bad result, unless i'm doing somethig wrong i was saying.
It was the case !
Doing this :

r = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$FF0000).ConvertToYV12(ChromaOutPlacement="MPEG2")
g = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$00FF00).ConvertToYV12(ChromaOutPlacement="MPEG2")

stackhorizontal(r,g)
o=stackvertical(last,last.fliphorizontal())

or this

r = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$FF0000).ConvertToYV16()
g = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$00FF00).ConvertToYV16()

stackhorizontal(r,g)
o=stackvertical(last,last.fliphorizontal())

was wrong !
The correct way was this :

r = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$FF0000)
g = blankclip(width=8,height=8,pixel_type="YV24",color_yuv=$00FF00)

stackhorizontal(r,g)
o=stackvertical(last,last.fliphorizontal()).ConvertToYV12(ChromaOutPlacement="MPEG2")

and with this correct way, finaly i've been able to verify the rigthness of the formula.

[PetitDragon]

:D:D:DNew version up, as usual, everything is on the first post.

Thanks jpsdr (and Myrsloik too:D). You are one.:thanks:

[Reel.Deel]

@jpsdr

What's your thoughts on releasing two binaries (x86/x64) that work with Windows XP and greater and do not require a specific instruction set (similar to tritical's version). I ask this because I think it could be a bit overwhelming to novice users. A friend of mine started using AviSynth not too long, he's a bit computer savvy but he's used to program installers doing all the work. Anyways, I told him to use your version of nnedi3, he got it to work but he was a bit confused due to all the binaries (7 for each platform :eek:). I helped him out and he now knows what instructions sets are. I'm not saying to get rid of the optimized binaries, more like have 2 downloads, the regular binaries for general users, and the optimized binaries for advanced users. Just some food for thought :).

One last thing, there's a tiny issue concerning the readme, csresize is not included in the syntax.

[jpsdr]

@jpsdr
What's your thoughts on releasing two binaries (x86/x64) that work with Windows XP and greater and do not require a specific instruction set (similar to tritical's version)

That's what you allready found in ReleaseXP directory. I'll check the readme, and if necessary, make it more clear.

@jpsdr
One last thing, there's a tiny issue concerning the readme, csresize is not included in the syntax.

Ah... Ok, i see, i'll update this later.

[feisty2]

I'm writing a floating point nnedi3 for vaporsynth, and I'm using your asm file for optimization
it works out pretty good, I failed to get the one coming with the original vaporsynth port to work, cuz that one is like, gcc style asm and I'm just NO GOOD at that kinda stuff
but still... I got a tiny little problem here :D
your "dotProd_SSE2" functions won't work here
I see you got "dotProd_m32_m16_SSE2" and "dotProd_m48_m16_SSE2", 2 things to replace "dotProd_C"
and I tried to replace "dotProd_C" with them "asize"-wise, like what you did, and the program gave me corrupted results
then I replaced "dotProd_C" with simply "dotProd_m48_m16_SSE2" and the program crashed.

any idea why? and how to fix?

[jpsdr]

First, just to be precise, technicaly it's not my functions, i've just put in outside files inline asm, to be able to make an x64 version, so, i personnaly didn't make them, and personnaly didn't do anything, it was allready like this.
If i can look at your code, i can check if maybe (a big "maybe") i see something odd.

Personnaly, i don't do things like Tritical has, playing with functions pointer or anything, i made different fonctions, and call them in different part of code.
If you go to my github and take a look at the code of my VDub filter, you'll see what i mean.

[feisty2]

Okay, and thx anyways, at least I can enjoy some of the SSE2 stuff from that file :)
Guess I'll have to mess with "dotprod" some more and try to figure out what is going wrong

[jpsdr]

As you wish, but you can understand that without even being able to look at the code, i obviously can't tell you what may be wrong...

[feisty2]

Source code?, ooh, here https://github.com/IFeelBloated/NNEDI3SF/blob/master/NNEDI3SF_SSE2.cpp
I'd really appreciate your help :D

[jpsdr]

Stupid question : You are aware of these ?
https://github.com/jpsdr/NNEDI3/commit/19a21726b8747cd691b33297b5e24615d55ab86d
https://github.com/jpsdr/NNEDI3/commit/948a177fe197bf0284977c42a195d6d6411f9190

[feisty2]

Not sure if my asm file is outdated or not actually...
Checking it right now

[feisty2]

@jpsdr
I got the latest x64 asm file and got results like this
http://i.imgur.com/ismYhdP.png

and how I call those 2 dotprod functions


static void selectFunctions(nnedi3sfData *d) {
const int asize = d->asize;
d->copyPad = copyPad<float>;
d->evalFunc_0 = evalFunc_0<float>;
d->evalFunc_1 = evalFunc_1<float>;

// evalFunc_0
d->processLine0 = processLine0_C<float>;

d->readPixels = pixel2float48_C<float>;
d->computeNetwork0 = computeNetwork0_SSE2;

// evalFunc_1
d->wae5 = weightedAvgElliottMul5_m16_SSE2;

d->extract = extract_m8_C<float, double, double>;
d->dotProd = (asize % 48) ? dotProd_m32_m16_SSE2 : dotProd_m48_m16_SSE2;

if ((d->fapprox & 12) == 0) { // use slow exp
d->expfunc = e2_m16_SSE2;
}
else if ((d->fapprox & 12) == 4) { // use faster exp
d->expfunc = e1_m16_SSE2;
}
else { // use fastest exp
d->expfunc = e0_m16_SSE2;
}
}


I'm doing it right... right?
then maybe the asm code is buggy?

[jpsdr]

then maybe the asm code is buggy?
That's not impossible, but on the other hand, since the time i've made the release, it should have poped up since, but maybe no one used this specific mode.
I don't have time right now, but first the pluggin i've made should be checked, playing with fapprox to be sure. If issues occure, original pluggin should be also tested, to see if error has been introduced.

[feisty2]

playing with fapprox to be sure. If issues occure, original pluggin should be also tested, to see if error has been introduced.

fapprox works fine (e0/1/2_m16_SSE2 all work out okay)
and I'm 85% sure they (dotprod_sse2 stuff) are broken, everything worked perfectly before I got them in.

but maybe no one used this specific mode.
seems so, dotProd_i16 is preferred over dotProd at int8 process, I removed dotProd_i16 cuz it won't work at higher precision. (0,4,8,12 are the only available fapprox values and 0,4,8,12 & 2 = 0, which means dotProd_i16 is disabled)

[Desbreko]

Seems the chroma shift doesn't get corrected with YV411.

g = BlankClip(length=1, width=16, height=16, pixel_type="YV24", color_yuv=$FF0000).KillAudio()
b = BlankClip(length=1, width=16, height=16, pixel_type="YV24", color_yuv=$00FF00).KillAudio()
StackVertical(StackHorizontal(g,b,g,b),StackHorizontal(b,g,b,g),StackHorizontal(g,b,g,b),StackHorizontal(b,g,b,g))
ConvertToYV411()
r16 = Dither_convert_8_to_16().Dither_resize16(Width()*2,Height()*2).DitherPost(mode=-1)
nn3 = nnedi3_rpow2(2, cshift="Spline36Resize")
Interleave(r16,nn3)

[jpsdr]

@Desbreko
According the thread talking about 420/444 convertion, i've allready linked several time in this thread, Y411 musn't be chroma shifted when resized according explaination from it, but the chroma shift from nnedi3 is corrected, try with and without cshift, and you'll see. So, i don't have the "16" stuff, but if the resize16 also shift chroma in YV411, according the information from the other thread, it shouldn't.
Edit : here (http://forum.doom9.org/showthread.php?p=1705237#post1705237).
Edit 2 : My mistake...
Edit 3 : Finaly, after re-checking again, no, the chroma shift of YV411 is correct, at least, according what explained in the link up.
Edit 4 : .... Ok, triple, quadruple, etc... check before i fall back on my feets again.

@feisty2
Original code is

if (fapprox&2) // use int16 dot products
{
if (opt==1) extract=extract_m8_i16_C;
else extract=extract_m8_i16_SSE2;
if (opt==1) dotProd=dotProdS_C;
else dotProd= (asize%48) ? dotProd_m32_m16_i16_SSE2 : dotProd_m48_m16_i16_SSE2;
}
else // use float dot products
{
if (opt==1) extract=extract_m8_C;
else extract=extract_m8_SSE2;
if (opt==1) dotProd=dotProd_C;
else dotProd= (asize%48) ? dotProd_m32_m16_SSE2 : dotProd_m48_m16_SSE2;
}

so playing with fapprox is necessary to switch the dotProd, after, i have no idea how to trig or not tne %48. I'm not and will not until a long time be under a 64 bit OS, so i can't make tests on the x64 pluggin for now, to see if problems occurs also. After, it's a standard chain of process, if problems occurs on the x64, test the x86, and if occurs also on the x86, test the original.

[feisty2]

okay, I'll install an x86 avs+ and report back :)

[Desbreko]

That post is saying the formula for correcting the chroma shift isn't the same for 4:1:1, not that it shouldn't be corrected at all. 4:1:1 is 1/4 horizontally subsampled, so instead of the 0.25*(1-Width_in/Width_out) formula for correcting 1/2 horizontally subsampled formats like YV12 and YV16, you have to use the formula 0.375*(1-Width_in/Width_out).

And nnedi3 itself doesn't cause horizontal chroma shift, provided it's used correctly. What's corrected when you set the cshift parameter is the center shift, which is a shift in both luma and chroma together. It's the resizing used to correct the center shift that causes the chroma shift. So of course there's no chroma shift without setting cshift.

[jpsdr]

That post is saying the formula for correcting the chroma shift isn't the same for 4:1:1, not that it shouldn't be corrected at all.

That's what i've begin to wonder, and finaly i've got the same conclusion, you're right.

Will modify this in the next release.

I'll wait for feisty2 results before doing a next release.

@feisty2
If you can provide the avisynth script of the post #188, it may help.

[feisty2]

weird, I got avs+ x64 and nnedi3.dll from NNEDI3_v0_9_4_17.7z/x64/Release_W7 and tried this

# Y8 clip #
nnedi3 (field=0,dh=True,fapprox=12,pscrn=1,nsize=3,nns=4,qual=2,etype=1)

and things seemed, pretty normal, nothing crappy happened

and this gave shits like #188

import vapoursynth as vs
core = vs.get_core()
#clp = GRAYS whatever
clp = core.nnedi3sf.nnedi3 (clp,field=0,dh=True,fapprox=12,pscrn=1,nsize=3,nns=4,qual=2,etype=1)
clp.set_output ()


guess I'll investigate dotProd more thoroughly and try to trace the problem...

[jpsdr]

New version, see first post.

[jpsdr]

New new version again... :D

[Groucho2004]

A couple of things about your builds that seem peculiar:

Since XP does not support AVX, I'm a bit puzzled about this version:
Release_Intel_XP_Core2_AVX2

Similarly in your VDub Filters:
Release_Intel_XP_Core2_AVX
Release_Intel_XP_Core2_AVX2

And, as expected, these versions crash on XP => "Illegal instruction"


Also, why do you create "XP" and "W7" versions although there is no code in nnedi3 (as far as I know) that uses APIs exclusive to Windows 6.x?

[jpsdr]

First, i've always thought than support of instruction was CPU only, it's the first time i see it can be OS dependant, and having an "illegall instruction" is possible even on a CPU which is capable of doing it, so on which it's not normaly/theorically illegal. Well, you can learn all the time.

For the "XP"/"W7", i just compile with "v120" or "v120_xp", no more, no less, to be sure, just in case it may have an effect, of compiling with "v120" produce something not working under XP.

I'll remove XP/AVX in future releases, if not working (i don't have CPU with AVX, even less AVX2, my best is SSE4.2, so never been able to test...).

[jpsdr]

New version.