nnedi2/nnedi3/eedi3

[markanini]

It's a noticeable improvement.

[jpsdr]

@tritical : Do you plan to update the 64bits version with ABS ?

[tritical]

Yes, but I'm thinking about changing it so that the weights are not compiled into the binary. There would be a string that points to a weight file. Then I don't have to have multiple versions of a 6MB dll on my web space (which is pretty limited) - only one copy of each set of weights. Although half of that 6MB size is due to the nns=256 weights, so I'm considering just dropping the 256 setting since it is too slow for video and does not offer much visual improvement over 128... metric wise the improvement is the same as 64->128. Also, the training takes a LONG time.

[Gser]

Originally Posted by tritical

Yes, but I'm thinking about changing it so that the weights are not compiled into the binary. There would be a string that points to a weight file. Then I don't have to have multiple versions of a 6MB dll on my web space (which is pretty limited) - only one copy of each set of weights. Although half of that 6MB size is due to the nns=256 weights, so I'm considering just dropping the 256 setting since it is too slow for video and does not offer much visual improvement over 128... metric wise the improvement is the same as 64->128. Also, the training takes a LONG time.

I have web space to spare if you are in need.

[Archimedes]

Thanks for the update, tritical. Indeed an improvement.

Source (400x300) >>

Spline36Resize(1600, 1200) >>
nnedi2_rpow2(rfactor=4, nsize=2, qual=2, cshift="Spline36Resize") >>
nnedi3_rpow2(rfactor=4, nsize=0, nns=4, qual=2, cshift="Spline36Resize"), SQR >>
nnedi3_rpow2(rfactor=4, nsize=0, nns=4, qual=2, cshift="Spline36Resize"), ABS >>

[jpsdr]

@tritical : Any news about a final version (32 & 64 bits) of nnedi3 with abs ?

[mandarinka]

BTW It seems there is no nnedi3 (for non-sse2 cpus) in the old_stuff folder.

[Sapo84]

I've been toying with nnedi3 (abs version) for the past days and I've found a somewhat strange behaviour.

When using

nnedi3_rpow2(rfactor=2,nns=2,nsize=3,cshift="Spline36Resize")

I think the filter is not doing the chroma shift part quite correctly.
In fact I think the filter actually does something very similar to

nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))

Which is obviously not correct, because the correct version should end with something like

MergeChroma(b.Spline36Resize(OX,OY,-1.00,-1.00,OX*2,OY*2))

If I'm correct the filter probably needs some kind of correction.

On another note it would be helpful (to me, at least) if nnedi3_rpow2 had more parameters, I'd like to have the option to disable the chroma planes processing and to skip the chroma shift, I know it can be easily done with two nnedi3 calls, but it would help in writing clearer scripts.

[Gavino]

Originally Posted by Sapo84

I think the filter is not doing the chroma shift part quite correctly.
In fact I think the filter actually does something very similar to
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))

Which is obviously not correct, because the correct version should end with something like
MergeChroma(b.Spline36Resize(OX,OY,-1.00,-1.00,OX*2,OY*2))

Why do you think that is not correct?
Avisynth assumes MPEG-2 sampling, so chroma pixels are vertically aligned with luma.
Hence, the horizontal shift correction is the same (-0.5) for both luma and chroma, and it is only the vertical one which differs (-0.5 for luma and -1.0 for chroma).

[Sapo84]

Originally Posted by Gavino

Why do you think that is not correct?
Avisynth assumes MPEG-2 sampling, so chroma pixels are vertically aligned with luma.
Hence, the horizontal shift correction is the same (-0.5) for both luma and chroma, and it is only the vertical one which differs (-0.5 for luma and -1.0 for chroma).

Well, that may be true (and in fact, I have no idea why that should not work), but then it means there's something fishy going on.

UToY() applied to the clip before processing
http://www.cbland.net/images4/01azazel.png

UToY() applied to the clip after -0.5 -1.0 chroma resize
http://www.cbland.net/images4/01azazel050100.png

UToY() applied to the clip after -1.0 -1.0 chroma resize
http://www.cbland.net/images4/01azazel100100.png

The second one is different from 1 and 3 (which are basically the same), that'd why I assumed -1.0 -1.0 was the way to go.

Edit: Did try with a different source, same problem.

[Gavino]

Originally Posted by Sapo84

UToY() applied to the clip before processing
http://www.cbland.net/images4/01azazel.png

UToY() applied to the clip after -0.5 -1.0 chroma resize
http://www.cbland.net/images4/01azazel050100.png

UToY() applied to the clip after -1.0 -1.0 chroma resize
http://www.cbland.net/images4/01azazel100100.png

The second one is different from 1 and 3 (which are basically the same), that'd why I assumed -1.0 -1.0 was the way to go.

Can you clarify the exact scripts used to produce those three results, please.
I can't see any difference visually between them, but using Compare shows they are all different, with 2 and 3 being (by a small margin) the most similar.
1 v 2: PSNR=51.99dB
1 v 3: PSNR=53.98dB
2 v 3: PSNR=55.88dB

[Sapo84]

Originally Posted by Gavino

Can you clarify the exact scripts used to produce those three results, please.

UToY()
___________
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))
UToY()
___________
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-1.0,-1.00,OX*2,OY*2))
UToY()

Originally Posted by Gavino

I can't see any difference visually between them, but using Compare shows they are all different, with 2 and 3 being (by a small margin) the most similar.

Well, the difference is 1/4 of a pixel, you need to zoom in A LOT.
I did a quick crop and zoom.
http://www.cbland.net/images4/Azazel_01.png
http://www.cbland.net/images4/Azazel_02.png
http://www.cbland.net/images4/Azazel_03.png

While 1 and 3 are pretty much identical the 2 suffer from a little shift, and it's the one with the (-0.5,-1) chroma shift, which should be the correct one.

Edit: After a bit of thinking I may have the reason why -1 -1 is correct, in fact nnedy does never resize horizontally, it just does Turnleft() and then Turnright(), so the chroma does get shifted by 1 pixel in both directions.

[Gavino]

Are OX and OY the original clip width and height?
So you are upsizing with nnedi3 and downsizing back to the original size?
I'm confused about your original point now. Where does nnedi3_rpow2 come into it?

Originally Posted by Sapo84

After a bit of thinking I may have the reason why -1 -1 is correct, in fact nnedy does never resize horizontally, it just does Turnleft() and then Turnright(), so the chroma does get shifted by 1 pixel in both directions.

But because of the difference between vertical and horizontal chroma/luma alignment (MPEG-2 sampling), there is no chroma shift from the horizontal resize. See the discussion starting at post #12 of this thread. As I said in post #14:

Originally Posted by Gavino

for the horizontal resizings, you need to use field=0 if doing TurnLeft.Nnedi2().TurnRight() and field=1 for TurnRight().Nnedi2().TurnLeft(). In either of these cases, there will be no shift at all.

[Sapo84]

Originally Posted by Gavino

Are OX and OY the original clip width and height?

Yeah, I should have wrote a.width() and a.heigth() for clarity sake.

Originally Posted by Gavino

So you are upsizing with nnedi3 and downsizing back to the original size?
I'm confused about your original point now. Where does nnedi3_rpow2 come into it?

nnedi3_rpow2(rfactor=2) and then resizing back to the original resolution is equal to the second script.
And I'm finding the chroma is shifted by 1/4 of a pixel (also I would like to have more control on the parameters, because I'd like to write a script like naa() and I would like to skip chroma processing (if I don't want to process chroma) or chroma shifting (I would like to do it afterwards), but these are just feature request).

Originally Posted by Gavino

But because of the difference between vertical and horizontal chroma/luma alignment (MPEG-2 sampling), there is no chroma shift from the horizontal resize. See the discussion starting at post #12 of this thread. As I said in post #14:

But horizontal chroma is never touched, nnedi only does vertical doubling, and vertical doubling has even shift (chroma line is not aligned vertically).
(well, that depends on what TurnLeft() and TurnRight() actually do to the chroma plane)

[Didée]

If anything can beat theory, it is practice.

loadplugin("nnedi3.dll")

r = blankclip(width=64,height=64,pixel_type="YV12",color_yuv=$FF0000)
g = blankclip(width=64,height=64,pixel_type="YV12",color_yuv=$00FF00)

stackhorizontal(r,g)
stackvertical(last,last.fliphorizontal())
o=last

nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)

interleave(o,last)

return(last)

To my eyes, that's a horizontal chroma shift, indeed.

[Gavino]

Originally Posted by Sapo84

(well, that depends on what TurnLeft() and TurnRight() actually do to the chroma plane)

TurnLeft and TurnRight preserve all original pixels, just re-arranging their positions. Hence the combination TurnRight().Nnedi3(1, dh=true).TurnLeft() does not introduce a chroma shift.

Strictly speaking, you could argue that TurnLeft and TurnRight should resample chroma but that would mean they were no longer lossless operations.

[Sapo84]

Originally Posted by Gavino

TurnLeft and TurnRight preserve all original pixels, just re-arranging their positions. Hence the combination TurnRight().Nnedi3(1, dh=true).TurnLeft() does not introduce a chroma shift.

Strictly speaking, you could argue that TurnLeft and TurnRight should resample chroma but that would mean they were no longer lossless operations.

Then I don't really know why in practice the chroma shift is there ^^''
Back to square one, I guess.

Originally Posted by Didée

To my eyes, that's a horizontal chroma shift, indeed.

Exactly.

Edit: Also

nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(128,128,-0.5,-0.5,256,256)MergeChroma(a.Spline36Resize(128,128,-1.0,-1.0,256,256))

when used instead of nnedi3_rpow2 does not introduce any chroma shift.

[tritical]

Gavino or Didée, any other thoughts on the horizontal shifting?

In terms of mpeg2 chroma placement nnedi2_rpow2() does not introduce a horizontal shift of chroma relative to luma... which to me can only mean that Avisynth's resizers do not assume mpeg2 chroma placement. I have not looked at the code to confirm this, but if it is the case then chaining many calls of nnedi3 (which assumes mpeg2 placement) with one of Avisynth's resizers (which assume mpeg1 maybe?) together would produce a shift as shown by Didée's script. However, if I change these lines of Didée's script:

nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)

to

nnedi3_rpow2(2).converttorgb24().spline36resize(128,128,-0.5,-0.5,256,256).convertbacktoyuy2().converttoyv12()

there is no horizontal chroma shift. Which I believe confirms my suspicion as converttorgb24() + convertbacktoyuy2().converttoyv12() uses mpeg2 placement.


Anyway, I'm planning to make a new release of nnedi3 with significantly faster prescreening + squared/abs weights in the same binary (with a parameter to switch between) and would like to get this issue cleared up.

EDIT: after looking at the code it is indeed what I suspected. Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma. I'll adjust nnedi3 to take this into account.

[poisondeathray]

Originally Posted by tritical

EDIT: after looking at the code it is indeed what I suspected. Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma. I'll adjust nnedi3 to take this into account.

Does this apply to all versions of avisynth , or were there some changes made to 2.6 alpha 3 that might change this behaviour ?

[Gavino]

Originally Posted by tritical

Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma.

Hmm, you're right.
I don't know why this hasn't occurred to me before - it's because the chroma sampling grid is not horizontally centred and is similar in nature to the problem in Bob that I deduced was present with interlaced YV12.

I don't know whether this is a performance shortcut, an oversight, or (dare I say it) a bug.

Originally Posted by poisondeathray

Does this apply to all versions of avisynth , or were there some changes made to 2.6 alpha 3 that might change this behaviour ?

It applies to all previous versions too.
The chroma is shifted right when upsizing and left when downsizing, and the amount depends on the scale ratio - probably imperceptible in most practical cases, but it definitely exists.