nnedi2/nnedi3/eedi3

[Mystery Keeper]

Ah. Completely different method. Now I see. Got to compare for quality/speed ratio.

[Boulder]

Has anybody else had stability issues with the latest version and qual or nns higher than the default values?

Using this script I get a reboot quite soon to the beginning:
MPEG2Source("2x01.d2v",cpu=4,moderate_h=25,moderate_v=45)
#SetMTMode(2)
#ChangeFPS(last,last,true)
AssumeTFF()
Crop(24,8,-20,-4,true)
ColorMatrix(d2v="2x01.d2v",threads=1,interlaced=true)
r=YADIFMod(mode=1,edeint=NNEDI3(field=-2,qual=2,threads=1))
d=r.Blur(1.58)
SRestore(r,mode=-2,frate=29.97,thresh=22,cache=10,dclip=d)
The previous version works fine.

The problem is that the rebooting points to an unstable system but my system is 24/7 Prime95-stable. Rebooting is also a problem because it doesn't leave any tracks to follow.

[yup]

Boulder!
I try
nnedi3(field=-2,nsize=3,qual=2)
for 2.58 official and Set 2.6 build without problem.
yup.

[Boulder]

Which CPU do you have? I have an Intel E6750.

EDIT: using opt=1 doesn't work either :(

[Didée]

Is the CPU actually OC'ed? You know - if yes, you need to cross-check with Stock settings. "Prime stable" is only a hint, in last instance it doesn't mean much. You can only prove instability, but you can not prove stability. I've made that experience too - with some certain OC settings (probably borderline), the system was 24h Prime and LinX stable. Still, very-complex Avisynth scripts tended to crash occasionally. Which disappeared when I relaxed the OC just a little bit.

[Boulder]

It's possible that it is due to overclocking.. Funny thing that during the three years that I've had the system as it is, there have not been such problems. Possibly the latest NNEDI3 just tickles a certain register the right way to cause issues :)

[yup]

Hi all!
How can upscale YUY2 source
ConvertToRGB24(interlaced=false)
nnedi3_rpow2(rfactor=2,cshift="spline36resize")
ConvertBackToYUY2(interlaced=false)
or using separate plane for Y,U and V.
Please advice.
yup.

[Didée]

Why jump through loops when it's not necessary?

INFO:

nnedi3 [...] works with YV12, YUY2, and RGB24 input.

[yup]

Didée!
May be problem with Set 2.6 build. My script crash if I using nnedi3_rpow2 for YUY2 colorspace.
For YV12 work fine.
Script simple

SetMTMode(2,4)
DirectShowSource("bbb.avi")
ConvertToYUY2(interlaced=true)# for ConvertToYV12(interlaced=true) work fine
AssumeTFF()
QTGMC()#YUY2 version
Crop(16,16,-16,-16)
nnedi3_rpow2(rfactor=2,cshift="spline64resize",fwidth=720,fheight=576)
yup.

2.58 official release work without problem.

[tritical]

I put up a version of nnedi3 with weights trained to minimize absolute error instead of squared error (see first post). At the moment the following nns/nsize combinations are missing new weights (they are still using the squared error trained weights) nns=256 @ 16x6, 32x6, 48x6, and 32x4. The absolute error trained weights seem to be better for image enlargements - they eliminate some artifacts currently produced by nnedi3.

The castle image of Archimedes:

http://forum.doom9.org/showthread.php?p=1418087#post1418087
nnedi3_rpow2(rfactor=4,nsize=0,nns=4,qual=2,cshift="Spline36Resize") - ABS (http://bengal.missouri.edu/~kes25c/castle_abs.png)
spline36resize(width*4,height*4) (http://bengal.missouri.edu/~kes25c/castle_spline36.png)

Lighthouse image:

nnedi3_rpow2(rfactor=4,nsize=0,nns=4,qual=2,cshift="Spline36Resize") - SQUARE (http://bengal.missouri.edu/~kes25c/lighthouse_square.png)
nnedi3_rpow2(rfactor=4,nsize=0,nns=4,qual=2,cshift="Spline36Resize") - ABS (http://bengal.missouri.edu/~kes25c/lighthouse_abs.png)
spline36resize(width*4,height*4) (http://bengal.missouri.edu/~kes25c/lighthouse_spline36.png)

Nothing huge, but some difference. I did this just to see what would happen since it only requires running the training program and no actual work :). Am going to try minimizing sqrt(abs(error)) next. Based on some initial tests on a small amount of training data I think it may offer more improvements for image enlargement.

[markanini]

It's a noticeable improvement. :)

[jpsdr]

@tritical : Do you plan to update the 64bits version with ABS ?

[tritical]

Yes, but I'm thinking about changing it so that the weights are not compiled into the binary. There would be a string that points to a weight file. Then I don't have to have multiple versions of a 6MB dll on my web space (which is pretty limited) - only one copy of each set of weights. Although half of that 6MB size is due to the nns=256 weights, so I'm considering just dropping the 256 setting since it is too slow for video and does not offer much visual improvement over 128... metric wise the improvement is the same as 64->128. Also, the training takes a LONG time.

[Gser]

Yes, but I'm thinking about changing it so that the weights are not compiled into the binary. There would be a string that points to a weight file. Then I don't have to have multiple versions of a 6MB dll on my web space (which is pretty limited) - only one copy of each set of weights. Although half of that 6MB size is due to the nns=256 weights, so I'm considering just dropping the 256 setting since it is too slow for video and does not offer much visual improvement over 128... metric wise the improvement is the same as 64->128. Also, the training takes a LONG time.

I have web space to spare if you are in need.

[Archimedes]

Thanks for the update, tritical. Indeed an improvement.

Source (400x300) >> (http://img263.imageshack.us/i/sn850652.png/)

Spline36Resize(1600, 1200) >> (http://img13.imageshack.us/i/sn850652spline36resize.jpg/)
nnedi2_rpow2(rfactor=4, nsize=2, qual=2, cshift="Spline36Resize") >> (http://img508.imageshack.us/i/sn850652nnedi2.jpg/)
nnedi3_rpow2(rfactor=4, nsize=0, nns=4, qual=2, cshift="Spline36Resize"), SQR >> (http://img13.imageshack.us/i/sn850652nnedi3sqr.jpg/)
nnedi3_rpow2(rfactor=4, nsize=0, nns=4, qual=2, cshift="Spline36Resize"), ABS >> (http://img851.imageshack.us/i/sn850652nnedi3abs.jpg/)

[jpsdr]

@tritical : Any news about a final version (32 & 64 bits) of nnedi3 with abs ?

[mandarinka]

BTW It seems there is no nnedi3 (for non-sse2 cpus) in the old_stuff folder.

[Sapo84]

I've been toying with nnedi3 (abs version) for the past days and I've found a somewhat strange behaviour.

When using

nnedi3_rpow2(rfactor=2,nns=2,nsize=3,cshift="Spline36Resize")

I think the filter is not doing the chroma shift part quite correctly.
In fact I think the filter actually does something very similar to

nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))

Which is obviously not correct, because the correct version should end with something like

MergeChroma(b.Spline36Resize(OX,OY,-1.00,-1.00,OX*2,OY*2))

If I'm correct the filter probably needs some kind of correction.

On another note it would be helpful (to me, at least) if nnedi3_rpow2 had more parameters, I'd like to have the option to disable the chroma planes processing and to skip the chroma shift, I know it can be easily done with two nnedi3 calls, but it would help in writing clearer scripts.

[Gavino]

I think the filter is not doing the chroma shift part quite correctly.
In fact I think the filter actually does something very similar to
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))

Which is obviously not correct, because the correct version should end with something like
MergeChroma(b.Spline36Resize(OX,OY,-1.00,-1.00,OX*2,OY*2))
Why do you think that is not correct?
Avisynth assumes MPEG-2 sampling (http://avisynth.org/mediawiki/Sampling#mpeg-1_versus_mpeg-2_sampling), so chroma pixels are vertically aligned with luma.
Hence, the horizontal shift correction is the same (-0.5) for both luma and chroma, and it is only the vertical one which differs (-0.5 for luma and -1.0 for chroma).

[Sapo84]

Why do you think that is not correct?
Avisynth assumes MPEG-2 sampling (http://avisynth.org/mediawiki/Sampling#mpeg-1_versus_mpeg-2_sampling), so chroma pixels are vertically aligned with luma.
Hence, the horizontal shift correction is the same (-0.5) for both luma and chroma, and it is only the vertical one which differs (-0.5 for luma and -1.0 for chroma).

Well, that may be true (and in fact, I have no idea why that should not work), but then it means there's something fishy going on.

UToY() applied to the clip before processing
http://www.cbland.net/images4/01azazel.png

UToY() applied to the clip after -0.5 -1.0 chroma resize
http://www.cbland.net/images4/01azazel050100.png

UToY() applied to the clip after -1.0 -1.0 chroma resize
http://www.cbland.net/images4/01azazel100100.png

The second one is different from 1 and 3 (which are basically the same), that'd why I assumed -1.0 -1.0 was the way to go.

Edit: Did try with a different source, same problem.

[Gavino]

UToY() applied to the clip before processing
http://www.cbland.net/images4/01azazel.png

UToY() applied to the clip after -0.5 -1.0 chroma resize
http://www.cbland.net/images4/01azazel050100.png

UToY() applied to the clip after -1.0 -1.0 chroma resize
http://www.cbland.net/images4/01azazel100100.png

The second one is different from 1 and 3 (which are basically the same), that'd why I assumed -1.0 -1.0 was the way to go.
Can you clarify the exact scripts used to produce those three results, please.
I can't see any difference visually between them, but using Compare shows they are all different, with 2 and 3 being (by a small margin) the most similar.
1 v 2: PSNR=51.99dB
1 v 3: PSNR=53.98dB
2 v 3: PSNR=55.88dB

[Sapo84]

Can you clarify the exact scripts used to produce those three results, please.
UToY()
___________
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-0.5,-1.00,OX*2,OY*2))
UToY()
___________
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(OX,OY,-0.5,-0.5,OX*2,OY*2)
MergeChroma(a.Spline36Resize(OX,OY,-1.0,-1.00,OX*2,OY*2))
UToY()

I can't see any difference visually between them, but using Compare shows they are all different, with 2 and 3 being (by a small margin) the most similar.
Well, the difference is 1/4 of a pixel, you need to zoom in A LOT.
I did a quick crop and zoom.
http://www.cbland.net/images4/Azazel_01.png
http://www.cbland.net/images4/Azazel_02.png
http://www.cbland.net/images4/Azazel_03.png

While 1 and 3 are pretty much identical the 2 suffer from a little shift, and it's the one with the (-0.5,-1) chroma shift, which should be the correct one.

Edit: After a bit of thinking I may have the reason why -1 -1 is correct, in fact nnedy does never resize horizontally, it just does Turnleft() and then Turnright(), so the chroma does get shifted by 1 pixel in both directions.

[Gavino]

Are OX and OY the original clip width and height?
So you are upsizing with nnedi3 and downsizing back to the original size?
I'm confused about your original point now. Where does nnedi3_rpow2 come into it?

After a bit of thinking I may have the reason why -1 -1 is correct, in fact nnedy does never resize horizontally, it just does Turnleft() and then Turnright(), so the chroma does get shifted by 1 pixel in both directions.
But because of the difference between vertical and horizontal chroma/luma alignment (MPEG-2 sampling), there is no chroma shift from the horizontal resize. See the discussion starting at post #12 of this thread. As I said in post #14:
for the horizontal resizings, you need to use field=0 if doing TurnLeft.Nnedi2().TurnRight() and field=1 for TurnRight().Nnedi2().TurnLeft(). In either of these cases, there will be no shift at all.

[Sapo84]

Are OX and OY the original clip width and height?
Yeah, I should have wrote a.width() and a.heigth() for clarity sake.

So you are upsizing with nnedi3 and downsizing back to the original size?
I'm confused about your original point now. Where does nnedi3_rpow2 come into it?
nnedi3_rpow2(rfactor=2) and then resizing back to the original resolution is equal to the second script.
And I'm finding the chroma is shifted by 1/4 of a pixel (also I would like to have more control on the parameters, because I'd like to write a script like naa() and I would like to skip chroma processing (if I don't want to process chroma) or chroma shifting (I would like to do it afterwards), but these are just feature request).

But because of the difference between vertical and horizontal chroma/luma alignment (MPEG-2 sampling), there is no chroma shift from the horizontal resize. See the discussion starting at post #12 of this thread. As I said in post #14:
But horizontal chroma is never touched, nnedi only does vertical doubling, and vertical doubling has even shift (chroma line is not aligned vertically).
(well, that depends on what TurnLeft() and TurnRight() actually do to the chroma plane)

[Didée]

If anything can beat theory, it is practice.

loadplugin("nnedi3.dll")

r = blankclip(width=64,height=64,pixel_type="YV12",color_yuv=$FF0000)
g = blankclip(width=64,height=64,pixel_type="YV12",color_yuv=$00FF00)

stackhorizontal(r,g)
stackvertical(last,last.fliphorizontal())
o=last

nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)
nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)

interleave(o,last)

return(last)

To my eyes, that's a horizontal chroma shift, indeed.

[Gavino]

(well, that depends on what TurnLeft() and TurnRight() actually do to the chroma plane)
TurnLeft and TurnRight preserve all original pixels, just re-arranging their positions. Hence the combination TurnRight().Nnedi3(1, dh=true).TurnLeft() does not introduce a chroma shift.

Strictly speaking, you could argue that TurnLeft and TurnRight should resample chroma but that would mean they were no longer lossless operations.

[Sapo84]

TurnLeft and TurnRight preserve all original pixels, just re-arranging their positions. Hence the combination TurnRight().Nnedi3(1, dh=true).TurnLeft() does not introduce a chroma shift.

Strictly speaking, you could argue that TurnLeft and TurnRight should resample chroma but that would mean they were no longer lossless operations.
Then I don't really know why in practice the chroma shift is there ^^''
Back to square one, I guess.


To my eyes, that's a horizontal chroma shift, indeed.
Exactly.

Edit: Also
nnedi3(1, dh=true,nns=2,nsize=3).turnright().nnedi3(1, dh=true,nns=2,nsize=3).turnleft()
a=last
Spline36Resize(128,128,-0.5,-0.5,256,256)MergeChroma(a.Spline36Resize(128,128,-1.0,-1.0,256,256))
when used instead of nnedi3_rpow2 does not introduce any chroma shift.

[tritical]

Gavino or Didée, any other thoughts on the horizontal shifting?

In terms of mpeg2 chroma placement nnedi2_rpow2() does not introduce a horizontal shift of chroma relative to luma... which to me can only mean that Avisynth's resizers do not assume mpeg2 chroma placement. I have not looked at the code to confirm this, but if it is the case then chaining many calls of nnedi3 (which assumes mpeg2 placement) with one of Avisynth's resizers (which assume mpeg1 maybe?) together would produce a shift as shown by Didée's script. However, if I change these lines of Didée's script:

nnedi3_rpow2(2,cshift="spline36resize",fwidth=128,fheight=128)

to

nnedi3_rpow2(2).converttorgb24().spline36resize(128,128,-0.5,-0.5,256,256).convertbacktoyuy2().converttoyv12()

there is no horizontal chroma shift. Which I believe confirms my suspicion as converttorgb24() + convertbacktoyuy2().converttoyv12() uses mpeg2 placement.


Anyway, I'm planning to make a new release of nnedi3 with significantly faster prescreening + squared/abs weights in the same binary (with a parameter to switch between) and would like to get this issue cleared up.

EDIT: after looking at the code it is indeed what I suspected. Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma. I'll adjust nnedi3 to take this into account.

[poisondeathray]

EDIT: after looking at the code it is indeed what I suspected. Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma. I'll adjust nnedi3 to take this into account.


Does this apply to all versions of avisynth , or were there some changes made to 2.6 alpha 3 that might change this behaviour ?

[Gavino]

Avisynth's resizers are assuming mpeg1 chroma placement when performing subpixel shifting on YV12/YUY2 - which results in a leftward shift of the chroma.
Hmm, you're right.
I don't know why this hasn't occurred to me before - it's because the chroma sampling grid is not horizontally centred and is similar in nature to the problem in Bob that I deduced was present with interlaced YV12.

I don't know whether this is a performance shortcut, an oversight, or (dare I say it) a bug.

Does this apply to all versions of avisynth , or were there some changes made to 2.6 alpha 3 that might change this behaviour ?
It applies to all previous versions too.
The chroma is shifted right when upsizing and left when downsizing, and the amount depends on the scale ratio - probably imperceptible in most practical cases, but it definitely exists.

[IanB]

Ahhhh! gawd more chroma positioning bugs. At first glance it seems to me the code dumbly assumes the chroma is positioned top left of the macrocell, not mpeg1 :confused:

The HResize code :- if (vi.IsYUY2()) {
pattern_chroma = func->GetResamplingPatternYUV(
vi.width >> 1,
subrange_left / 2,
subrange_width / 2,
target_width >> 1,
false, tempUV, env );
}
else if (vi.IsPlanar() && !vi.IsY8()) {
const int shift = vi.GetPlaneWidthSubsampling(PLANAR_U);
const int div = 1 << shift;

pattern_chroma = func->GetResamplingPatternYUV(
vi.width >> shift,
subrange_left / div,
subrange_width / div,
target_width >> shift,
true, tempY, env );
}

The VResize code :- if (vi.IsPlanar() && !vi.IsY8()) {
const int shift = vi.GetPlaneHeightSubsampling(PLANAR_U);
const int div = 1 << shift;

resampling_patternUV = func->GetResamplingPatternRGB(
vi.height >> shift,
subrange_top / div,
subrange_height / div,
target_height >> shift,
env);
}

The centre correction code in the Resampling Pattern generators :- // the following translates such that the image center remains fixed
double pos;
...
if (fir_filter_size == 1) // PointResize
pos = subrange_start;
else
pos = subrange_start + ((subrange_width - target_width) / (target_width*2));

[tritical]

Hopefully Gavino will explain what it should be (and why upsizing results in right shift and downsizing results in left) because after thinking about it some more I realized that my initial thoughts were not correct.. and after thinking it about it a little longer I am just more confused :p.

So for right now I am releasing v0.9.3 of nnedi3, which has the new prescreener and abs/square weights (switch with 'etype' parameter). It is linked in the first post. I will fix the chroma placement for YV12/YUY2 in the future... though it isn't really a problem unless you chain a bunch of calls together. This version is still missing a few abs weights for nns=4 (I think nsize=1/2/3). The new prescreener is more aggressive, but could be made even more aggressive since I can't tell any difference between it and the old prescreener at this point. Planning on addressing these issues and then releasing v1.0, as well as updating the 64-bit version.

[Gavino]

Hopefully Gavino will explain what it should be (and why upsizing results in right shift and downsizing results in left) because after thinking about it some more I realized that my initial thoughts were not correct.. and after thinking it about it a little longer I am just more confused :p.
I'm not sure yet what the solution is, but the way the shift happens is as follows.

The resizer core is designed to preserve the position of the image centre, and it does this for both luma and chroma independently.
This works fine for vertical resizing, since in that case the centres of the luma and chroma sampling grids coincide for all formats (including YV12, where the chroma samples are vertically positioned between the luma samples).

But for horizontal resizing (except YV24), the placement is such that the luma and chroma centres do not coincide - in each case the chroma centre is to the left of the luma centre.
Eg for YV12 and YUY2:
luma centre
|
Luma: L L L L ... L L L L ... L L L L
Chroma: C C ... C C ... C C
|
chroma centre
Since the distance between the two centres is proportional to pixel spacing, it changes on a resize.
For upsizing, pixel spacing decreases, so the chroma centre moves closer to the luma centre (hence to the right), and conversely for downsizing it moves to the left. Since the output chroma is calculated based on a fixed centre, the visual result is a corresponding chroma shift, right for upsizing and left for downsizing.

We can quantify the shift as follows.
For YV12 and YUY2, the chroma centre is 0.5 luma pixels to the left of the luma centre.
When resizing from width Win to Wout, output pixel spacing in terms of input is multiplied by Win/Wout.
Hence the net (rightwards) chroma shift is a distance of 0.5*(1-Win/Wout) input luma pixels.
For a 2x upsize, this equals 0.25, for a 2x downsize it's -0.5 (ie 0.5 to the left).

Similarly, for YV411, the net shift is 1.5*(1-Win/Wout), ie 3 times as much as for YV12/YUY2.

[IanB]

So we need to maintain the absolute pixel offset of chroma centre to luma centre, i.e. 0.5 pixels left for YV12/YV16/YUY2 and 1.5 pixels left for YV411.

Structurally it may help to move the centre offset calculation up a level into the [HV]Resize constructors so as to avoid fighting that calculation in the generator.

[Gavino]

Structurally it may help to move the centre offset calculation up a level into the [HV]Resize constructors so as to avoid fighting that calculation in the generator.
Yes, I was thinking along those lines too.

[Gavino]

So we need to maintain the absolute pixel offset of chroma centre to luma centre, i.e. 0.5 pixels left for YV12/YV16/YUY2 and 1.5 pixels left for YV411.
I think that's the end effect, but an easier way to look at it is that for horizontal resize, we want the first pixel to have the same absolute offset for both luma and chroma.

I think the following should work.
First remove the centre adjustment code from the resample_functions.
Then for FilteredResizeH, it is simply a matter of inserting at the start:
if (fir_filter_size != 1) // preserve image centre, except for PointResize
subrange_start += ((subrange_width - target_width) / (target_width*2));
For FilteredResizeV, you need to treat YV12 specially (changes in blue):
if (vi.IsRGB())
subrange_top = vi.height - subrange_top - subrange_height;

// offset to preserve image centre, except for PointResize
const double offset = (fir_filter_size == 1) ? 0 : ((subrange_width - target_width) / (target_width*2));

resampling_pattern = func->GetResamplingPatternRGB(vi.height, subrange_top+offset, subrange_height, target_height, env);

if (vi.IsYV12()) {
resampling_patternUV = func->GetResamplingPatternRGB(
vi.height >> 1,
subrange_top / 2 + offset,
subrange_height / 2,
target_height >> 1,
env);
}
else if (vi.IsPlanar() && !vi.IsY8()) {
const int shift = vi.GetPlaneHeightSubsampling(PLANAR_U);
const int div = 1 << shift;

resampling_patternUV = func->GetResamplingPatternRGB(
vi.height >> shift,
(subrange_top+offset) / div,
subrange_height / div,
target_height >> shift,
env);
}
What do you think?

[IanB]

I think you might be right, the offset for chroma needs to be the same as for luma with co-located chroma pixels (YUY2, mpeg2 YV12, etc).

But I think it is FilteredResizeH (Horizontal) that needs the special treatment.

[Gavino]

I think you might be right, the offset for chroma needs to be the same as for luma with co-located chroma pixels (YUY2, mpeg2 YV12, etc).

But I think it is FilteredResizeH (Horizontal) that needs the special treatment.
It's true that FilteredResizeH is the one that fails with the existing code. However, the real special case is YV12 vertical resize, since all the others have co-located chroma.

By moving the centre correction up to the constructors, it can be calculated always in units of luma pixels. Then the horizontal case becomes uniform and only YV12 vertical needs to be treated differently. (At least, I think so.)

I am assuming that other planar formats with vertically subsampled chroma would have co-located chroma. Is this correct? (As far as I know, we don't actually have any yet.)

[IanB]

Okay, with the existing code.

The value of ((subrange_width - target_width) / (target_width*2)) is independent of chroma scaling.

So the luma and chroma samplers get the same numerical correction value, i.e. the initial value of "pos".

This results in the chroma and luma centre points moving with respect to each other with the FilteredResizeH as explained in post #385 above. There is currently no consideration given to the co-location of the chroma positioning for YUY2, YV12 and YV411.

For the FilteredResizeV this operation is correct, but only because the chroma and luma centre points are coincident.

In summary, for co-located samples, i.e. YUY2, YV12 & YV411, the correction values needs to be scaled by the chroma subsampling to keep the centre points relative relationship. When the centre points are already coincident, i.e. vertical and mpeg1 then the same correction value needs to be applied.

[Gavino]

In summary, for co-located samples, i.e. YUY2, YV12 & YV411, the correction values needs to be scaled by the chroma subsampling to keep the centre points relative relationship. When the centre points are already coincident, i.e. vertical and mpeg1 then the same correction value needs to be applied.
Yep, that's it.

Are you also saying that vertically subsampled chroma is always centred in all planar formats? If so, my suggested code changes can be simplified. Also, I had inadvertently used subrange_ and target_width instead of _height in the earlier post. So my revised FilteredResizeV changes are:
if (vi.IsRGB())
subrange_top = vi.height - subrange_top - subrange_height;

// offset to preserve image centre, except for PointResize
const double offset = (fir_filter_size == 1) ? 0 : ((subrange_height - target_height) / (target_height*2));

resampling_pattern = func->GetResamplingPatternRGB(vi.height, subrange_top+offset, subrange_height, target_height, env);

if (vi.IsPlanar() && !vi.IsY8()) {
const int shift = vi.GetPlaneHeightSubsampling(PLANAR_U);
const int div = 1 << shift;

resampling_patternUV = func->GetResamplingPatternRGB(
vi.height >> shift,
subrange_top / div + offset,
subrange_height / div,
target_height >> shift,
env);
}

[tritical]

Put up version 0.9.4. Some more optimizations, plus more aggressive prescreener. I still can't tell the difference between the most aggressive new prescreener level (level 0) and the old prescreener. However, it is much much faster and flags a lot less pixels for the predictor nn. For small nsize and nns combinations where the prescreener computation was taking a significant chunk of the processing time in v0.9.2 the speedup is pretty good. Will work on fixing the chroma issues in nnedi3_rpow2 next. I'm probably going to add a parameter which specifies horizontal chroma placement.

[poisondeathray]

thanks!

[wOxxOm]

@tritical: is it possible to train/guide nnedi3 for such a low-contrast thin double lines like on the pocket slit below?

http://img42.imageshack.us/img42/7055/pocketslit.png

prescreener enabled (all levels) on the left vs disabled on the right.

[henryho_hk]

Put up version 0.9.4.

Thank you, tritical!

[aegisofrime]

Thanks tritical. Any possibility of a 64-bit build? :)

[Undead Sega]

Sorry for asking such a noobish question, but despite me actually using it recently, what is NNEDI3 ABS???

[tritical]

wOxxOm, could you upload a short sample of that video (a part that includes the frame you showed) somewhere? :thanks:

Undead Sega, nnedi3_abs is nnedi3 with predictor network weights trained to minimize l1 norm (absolute error) instead of the original training which minimized l2 norm (squared error).

[Undead Sega]

ohhh right, that's great I suppose...

Can you tell me exactly what does this mean by any chance? :)

[tritical]

The predictor networks in nnedi3 take in an area around the pixel in question, and, based on those pixel values, produce a guess for the missing pixel value. Those predictor networks are just mathematical models with sets of parameters (weights). To learn the parameters I take lots of training examples (grabbed from a 1000 frame video sequence containing images from many different sources), start with a random initial guess for the weights, and then perform online gradient descent training. Which simply means that I repeatedly grab a single training pattern, run the pattern through the model, compute the error in guessing the output value, compute the partial derivative of the error with respect to each weight (term) in the model, and then update the weights. Keep repeating until a minimum in the error surface is reached (progressive stops).

L1 (absolute error) vs L2 (squared error) simply define how I compute the error term. L2 will weight large errors more - an error of 20 will count 16 times as much as an error of 5 - whereas L1 will weight errors proportionally - an error of 20 will count 4 times as much as an error of 5. So with the L2 error measure the model will spend more 'effort' trying to fit the hardest cases since it will make more sense to fit the hard cases just a little better at the cost of some accuracy in a lot of the easy cases (for example, drop a single error from 100 to 20 while increasing many errors of 5 to 8, etc...). L1 will tip the balance more toward fitting the easier cases better. For deinterlacing, I think L1 makes more sense than L2 because after a certain point (say a miss of 20 or whatever) it doesn't matter if you miss the true value by 50 or 100 or 150 they will all look bad. However, the model will spend a lot of effort trying to fit the 150 case just a little better. On the other hand, if you are close to the true value a small difference can make a big visual improvement (for example: miss by 2 vs 10). The results of nnedi3_abs seem to back this up. In fact, I think going even further, using L0.5 - sqrt(abs(error)) - produces better results based on my initial tests.

[mandarinka]

Just an impression from me:
I was using qtgmc/srestore on a fieldblended anime dvd (ova from 1991), and I *think* I got better results from the old version ('etype=1' now I think). It was similar to that issue posted earlier - if there were paralel lines running close to each other, they would get less sharp and defined, compared to teh old method. I think there were some places where the new version did better but IIRC they weren'T that noticeable.

I didn't spot this watching of course, it was when comparing using interleave() - but how else is one suppossed to spot anything, right :)
As I said, it's jsut my impression from a brief testing on one source and through qtgmc. But if you are using that (and similarly, probably daa() with nnedi3 for antialiasing anime), you might want to run your own comparisons.