Vertical Distortion between 8-bit and 16-bit

[MysteryX]

There is a vertical distortion difference between these two scripts. The 8-bit version is missing a few lines at the bottom which the 16-bit version has. I'm not applying any cropping to the bottom. Why is this happening?

8-bit script (missing lines at the bottom, vertical distortion)

SetMTMode(3,8)
file="F:\NaturalGrounding\Temp\Preview.avi"
AviSource(file, audio=true, pixel_type="YV12")
SetMTMode(5)
KNLMeansCL(D=2, A=1, h=3, device_type="GPU")
SetMTMode(2)
Crop(0, 0, -8, -0)
Double="""edi_rpow2(2, nns=4, cshift="Spline16Resize", Threads=2)"""
SuperRes(2, 0.43, 0, Double, MatrixIn="601")
InterFrame(Cores=8, Tuning="Smooth", NewNum=60000, NewDen=1001, GPU=true)
Double="""edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=944, fheight=724, Threads=2)"""
SuperRes(2, 0.43, 0, Double, lsb_upscale=true)
Spline36Resize(940, 720, 0, 4, -4, -0)

Encoding result (https://www.spiritualselftransformation.com/files/media-encoder-new.mkv)

16-bit script (looks better)

SetMTMode(3,8)
file="F:\NaturalGrounding\Temp\Preview.avi"
AviSource(file, audio=true, pixel_type="YV12")
SetMTMode(5)
KNLMeansCL(D=2, A=1, h=3, device_type="GPU")
SetMTMode(2)
Crop(0, 0, -8, -0)
Double="""edi_rpow2(2, nns=4, cshift="Spline16Resize", Threads=2)"""
SuperRes(2, 0.43, 0, Double, MatrixIn="601")
InterFrame(Cores=8, Tuning="Smooth", NewNum=60000, NewDen=1001, GPU=true)
Double="""edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=944, fheight=724, Threads=2)"""
SuperRes(2, 0.43, 0, Double, lsb_upscale=true, lsb_out=true)
Dither_resize16(940, 720, 0, 4, -4, -0)
DitherPost()

Encoding result (https://www.spiritualselftransformation.com/files/media-encoder-new-10bit.mkv)

[Desbreko]

Avs's resizers and Dither_resize16 behave differently when src_width or src_height is set to 0. In the 8-bit script, either remove the -0 from the Spline36Resize call at the end or set it to 724 to make it match the 16-bit script. Or just use ResizeX instead of using Spline36Resize directly since its behavior matches Dither_resize16's, and it'll also avoid the chroma shift caused by Avs's resizers.

[MysteryX]

Chrome shift caused by Avs's resizers? What about NNEDI3's sub-pixel shifting and downscaling, what happens if I use ResizeX instead of BicubicResize?

Edit: Never mind, edi_rpow2 script does use ResizeX for sub-pixel shifting.

[MysteryX]

Now with ResizeX, I'm not losing lines at the bottom; but there's a band at the bottom that is darker. There's one version of my encoded files that has that dark band very narrow and it almost looks perfect but I can't reproduce how I did that.

There is perhaps a darker half-pixel in the original, but this band is much larger than that. What's going on here and how can I fix that?

First it was stretched too much, now it's the opposite... I think the 10-bit encoded video I uploaded has that dark band.

[Desbreko]

The way that you're shifting the image during the resize is duplicating the bottom row of pixels a few times, so if there's a narrow dark band along the bottom of the video to begin with, that's going to make it bigger.

[MysteryX]

Any way to avoid that?

[AzraelNewtype]

Crop before resizing instead of using those parameters as an ersatz crop. The problem is essentially that the sampling algorithm is looking outside the border you're supplying to calculate what the edge pixels are. Remove the pixels you don't want and they cannot impact the output.

[MysteryX]

The only reason I was doing cropping after is because CROP needs to be MOD 4 so I crop what I can first and then re-crop again after. But... perhaps I can just crop with more precision first by using resize's cropping instead of the crop command?

[StainlessS]

Crop off whatever you want to (EDIT: Restricted by colorspace requirements), then resize Mod 4.

[Desbreko]

Cropping is only restricted to mod2 for YV12, so at most you'll lose one row of clean pixels by cropping first. Or to clean up dark borders instead of cropping them, you could try using BalanceBorders or edgefix2.

Btw, I just fixed a bug in ResizeX that happened when src_height was negative, so you'll want to update your copy of the script.

[bxyhxyh]

Dither_resize16 don't crop when (src_left > 0 and src_width== 0) OR (src_top > 0 and scr_height == 0)
First one shifts video to left side and second one shifts up. Or both

To avoid that you shoud write something like
Dither_resize16(940, 720, left_crop_value, top_crop_value, source.width() - left_crop_value - right_crop_value, source.height() - top_crop_value - bottom_crop_value)

For your script write it as dither_resize16(940,720,0,4,last.width()-4,last.height()-4) This will resize correctly.

Edit: Or you can just crop before resize.
dither_crop16(0,4,-4,0)
dither_resize16(940,720)

[MysteryX]

Simply updating ResizeX seems to have solved the problem for the most part.

There is still some strange distortion at the bottom of the frame, but it's subtle.
http://s20.postimg.org/7ly0jzkrd/Bottom_Distortion.jpg (http://postimg.org/image/7ly0jzkrd/)

I tried applied EdgeFix2 and that didn't help. Any idea about that?

[Desbreko]

That's odd because the bug shouldn't have affected your script since you weren't using a negative src_height value.

BalanceBorders will probably work better on this source, but there's only so much either can do to clean up edges. You can try tweaking their settings, but if you can't get the edge clean enough with them, I'd just crop a couple pixels off.

[MysteryX]

I don't think I changed anything other than updating ResizeX and the problem is gone. I can crop 1 pixel at the bottom that is still darker and it looks fine. And then there's a larger band with wave patterns. The area of that distortion is the same as what got darker before updating ResizeX.

[MysteryX]

It turns out that wave distortion is in the source video, so everything is fine. It also appears that the x264 encoder mostly discards it.