Convert Stack16 Format / Problems with DitherTools

[MysteryX]

I'm getting to realize that with SuperRes, what gives the best result is this:
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")

In this case, the NNEDI3 subpixel shift and downscaling is done in Stack16 format with DitherTools. It is then converted back to YV12 before converting to 16-bit shader format.

This throws data away unnecessarily.

How can I go about converting Stack16 format? If I understand correctly, it can be any data format (YV12, YV24, RGB32) and it doubles the height of the video?

As part of my conversion process, I first convert YV12 data to YV24 using ConvertToYV24, and I then copy that data over into the new format. YV24 format in Stack16 would be simple to convert.

But what about YV12? I don't think calling ConvertToYV24() on YV12 Stack16 will give anything good. How can I then convert Stack16 from YV12 to YV24?

Another option might be to do something like
ConvertToYV24()
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")
ConvertToYV12()

I currently support converting to/from YV24, RGB24 and RGB32. All other formats must first be converted to those formats using standard conversion functions.

[MysteryX]

Converting from YV24 Stack16 into 8-bit stack format works with this
out[2] = y;
if (y2 >= 128)
out[2]++;
out[1] = u;
if (u2 >= 128)
out[1]++;
out[0] = v;
if (v2 >= 128)
out[0]++;

Convert to UINT16 data, however gives the right image except that it becomes green. Black become green, when I leave Y2=U2=V2=0. When those are set, it looks the same but weird black and green line patterns get added on top of the image.
char Buffer[6];
Buffer[0] = y;
Buffer[1] = y2;
Buffer[2] = u;
Buffer[3] = u2;
Buffer[4] = v;
Buffer[5] = v2;
uint16_t* pBuffer = (uint16_t*)&Buffer;

uint16_t *pOut = (uint16_t*)out;
pOut[0] = pBuffer[0];
pOut[1] = pBuffer[1];
pOut[2] = pBuffer[2];


This gets called like this

if (stack16)
height >>= 1;

unsigned char Y, U, V, Y2, U2, V2;
for (int y = 0; y < height; ++y) {
for (int x = 0; x < width; ++x) {
Y = py[x];
U = pu[x];
V = pv[x];

if (stack16) {
Y2 = py[x + pitch1Y * height];
U2 = pu[x + pitch1UV * height];
V2 = pv[x + pitch1UV * height];
convStack16(Y, U, V, Y2, U2, V2, dstLoop + (x << precisionShift));
}
else
convInt(Y, U, V, dstLoop + (x << precisionShift));
}
py += pitch1Y;
pu += pitch1UV;
pv += pitch1UV;
}


Any idea what's wrong here?

It also gave me an idea to get faster conversion from UINT16 to UINT8: take one byte and drop the other.

[MysteryX]

I got the conversion from YV24 Stack16 to Shader to work after reviewing the whole code.

What's left is to convert YV12 to YV24 in Stack16 format. How can I do this?

[wonkey_monkey]

Just a quick guess, without going hunting for convStack16, but is it because you haven't allowed for the fact that U and V are biased? They can't be treated as an unsigned char for maths purposes, as you can with Y, and you also can't treat as signed char either.

U and V are stored with "grey" as a value of (I think) 128. 0/0 actually represents U/V values of -128/-128, which is green.

[MysteryX]

char represents raw data; it doesn't matter what's in it.

Conversion is now working. I just want to know how I can convert from YV12, as currently the Stack16 conversion only supports YV24.

[MysteryX]

Here's another question about DitherTools. It's not working well in MT mode; it randomly crashes in a nasty way.

The documentation (http://avisynth.nl/index.php/Dither_tools) says
If you’re going multi-threaded using the MT command, make sure you have set splitvertical=true

Where do I have to set that?

I'm using edi_rpow2 which uses ResizeX which uses Dither_resize16. I don't see any such argument on it.

[MysteryX]

Ok. Conversion from YV24 DitherTools to 8-bit Shader is working. Converting to 16-bit Shader, however, adds distortion to the image.

SuperRes(2, .43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, threads=2)""", UpscaleStack16=true)

Within SuperRes.avsi, if I convert Stack16 with Precision=1 (UINT8)
http://s20.postimg.org/ts19xe0nt/Dither_To8.png (http://postimg.org/image/ts19xe0nt/)

if I convert with Precision=2 (UINT16), there is distortion.
http://s20.postimg.org/3l031fie1/Dither_To16.png (http://postimg.org/image/3l031fie1/)

The code doing the conversion is here
https://github.com/mysteryx93/AviSynthShader/blob/master/Src/ConvertToShader.cpp#L165

Any idea what I'm doing wrong? There really is no math whatsoever to bring the two separate bytes together. The only thing I could think of is getting the values from the wrong memory address, but that seems right.

[Reel.Deel]

Here's another question about DitherTools. It's not working well in MT mode; it randomly crashes in a nasty way.

The documentation (http://avisynth.nl/index.php/Dither_tools) says
If you’re going multi-threaded using the MT command, make sure you have set splitvertical=true

Where do I have to set that?



The splitvertical=true argument is only used in the old MT plugin (http://avisynth.nl/index.php/MT).

[Desbreko]

What's left is to convert YV12 to YV24 in Stack16 format. How can I do this?
Dither_resize16(Width(),Height()/2, csp="YV24")

[MysteryX]

Dither_resize16(Width(),Height()/2, csp="YV24")
That works. I suppose ConvertToYV12 is
Dither_resize16(Width(),Height()*2, csp="YV12")

What's left is to fix the distortion issue, and add support for RGB24/RGB32

[Desbreko]

No, to change only the chroma sampling, you always need to use Height()/2 regardless of the colorspace you're converting to because the target dimensions are the luma resolution, which doesn't change, and you're working with a stack16 clip which is double height.

[MysteryX]

I fixed a bug that got introduced in ConvertFromShader; but that didn't fix the distortion issue.

If I use this script, everything looks good
ImageSource("Lighthouse.png").ConvertToYV24()
edi_rpow2(2, nns=4, cshift="Spline16Resize", lsb=true, threads=2)
ConvertToShader(precision=2, Stack16=true)
ConvertFromShader(precision=2, format="YV12", Stack16=true)


I'm not seeing the extra Stack16 details to see whether they're good.

Either the bug is somewhere in the conversion, or somewhere in the way I handle the various textures with SuperRes.

To know that, I'd need to extrapolate the smaller details to have visual cues without using SuperRes.

dither_resize16 doesn't have a lsb_in argument... how can I run it on Stack16 data?

[Reel.Deel]

dither_resize16 doesn't have a lsb_in argument... how can I run it on Stack16 data?

It doesn't have it because dither_resize16 only works with stack16.

[MysteryX]

Oh. Got it.

This works, with or without ConvertToFloat. This means the problem isn't here.

ImageSource("Lighthouse.png").ConvertToYV24()
Upscale="""edi_rpow2(2, nns=4, cshift="Spline16Resize", lsb=true, threads=2)"""
Eval(Upscale)
#ConvertToShader(2, Stack16=true)
#ConvertFromShader(2, format="YV24", Stack16=true)
dither_resize16(int(width*1.5), int(height*1.5/2))
DitherPost()


Probably the SuperRes mask isn't being applied with the right proportions... shader arguments misconfigured. Had that problem before and had been pulling my hairs for a while. I guess it's there.

Edit: I really don't see where it could be wrong. Once I convert from Stack16, I don't have any clip with altered height.
https://github.com/mysteryx93/AviSynthShader/blob/master/Shaders/SuperRes/SuperRes.avsi

[MysteryX]

Found the problem. I'll be able to do comparison screenshots tomorrow.

[MysteryX]

RGB looks too complicated to convert; unless someone can lay it out in simple terms?

I don't personally need it for RGB anyway.

[bcn_246]

Tried LoadPlugin("C:\Standalone Programs\StaxRip v1.2.2.0\Applications\AviSynth plugins\ffms2\ffms2.dll")
FFVideoSource("C:\Users\Ben\Desktop\video.mkv", cachefile = "C:\Users\Ben\Desktop\video.ffindex")
ConvertToRGB24()
Upscale="""edi_rpow2(2, nns=4, cshift="Spline16Resize", lsb=true, threads=2)"""
Eval(Upscale)
#ConvertToShader(2, Stack16=true)
#ConvertFromShader(2, format="YV24", Stack16=true)
dither_resize16(int(width*1.5), int(height*1.5/2))
DitherPost() and got the following... have all the required plugins, am using AviSynth v2.6 (have tried the MT and non-MT versions). Also tried the last official NNEDI3 and the 0.9.4.5 mod. Also tried disabling dither (and losing the two lines) and replacing the dither-specific plugins with the general one. Keep getting this same error... any ideas?

http://i.imgur.com/zYtswYJ.png

Thanks,

Ben

[Desbreko]

You need jpsdr's nnedi3 v0.9.4.20 for edi_rpow2.

[bcn_246]

You need jpsdr's nnedi3 v0.9.4.20 for edi_rpow2.

Thanks, tried downloading and trying different versions (W7 SSE 4.2, XP SSE 4.2, plain W7, plain XP). Same error thrown each time. Have ResizeX and the associated plug-ins installed... using AVISynth 2.6.0 (MT 2015.02.20) have tried reverting to non-MT also, same error.

Just tried using nnedi3ocl, eedi2, and eedi3... under the edi=" perimeter, similar errors with all (two lines causing an error)

The lines that are causing the issue end up being...

nnedi3 line #629, eedi3 line #789
\ : input

nnedi3 line # 334, line eedi3 #433
\ : NOP()

Any ideas?

TIA

[Desbreko]

I just used the same script on a video of mine and it worked fine, so I'm not sure what the problem is. Do nnedi3 or any of the other edi plugins work by themselves?

[bcn_246]

Yep, alone all of the aforementioned work fine... the cshift for SuperResolution Checked and nnedi3_rpow2 works fine (have used YadifMod+NNEDI3 as my default deinterlacer, nnedi3_rpow2 as my default upscale and EEDI3 for really tricky content (with QTGMC). Are there any other dependencies I am missing? Using Windows 7 with a i7 4820 (definitely SSE 4.2 capable, and have tried the non SSE versions of NNEDI3). Might be worth backing up my plug in folder and filling it with the required avsi and dlls required for this script. Thanks again for your help. :)

[Desbreko]

It sounds like you should be good. Maybe try redownloading the edi_rpow2 script to make sure it didn't get messed up somehow? And yeah, emptying everything out of your plugins autoload folder except for what the script needs is also worth a shot. (In this case, edi_rpow2 should only require nnedi3, Dither, and ResizeX.)

[bcn_246]

It sounds like you should be good. Maybe try redownloading the edi_rpow2 script to make sure it didn't get messed up somehow? And yeah, emptying everything out of your plugins autoload folder except for what the script needs is also worth a shot. (In this case, edi_rpow2 should only require nnedi3, Dither, and ResizeX.)

Already tried redownloading the avsi for ResizeX and edi_rpow the media fire source, so I figure a clean plugin directory with only the required is my best bet (possible another filter is interfering). I will try and grab the code from one of the posts (think it was somewhere). Also are you using 2.6.0.5 MT? The lines that are causing issue seen to be native Avisynth ones (rather than specific to a particular filter).

Will let you know how it goes.

Thanks again, Ben :)

[Desbreko]

I use normal AviSynth 2.6, but I don't think it should make a difference for this.

[bcn_246]

I use normal AviSynth 2.6, but I don't think it should make a difference for this.

Okay, when I get home I will try cleaning out the plugins (rename plugin directory, reinstall Avisynth v2.6). Can you tell me which avisynth.dll version you are using? I be barking up the wrong tree, but as of the lines that throw the error appear to be internal Avisynth commands it is possible...

I will try first just installing just the required EDI dependencies, but without Dither (and use the lsb=false value), I will then try with Dither installed.

Way to often I have seen this kind of issue on here, a script works fine for one person but for another it doesn't, 99% of the time it is that one is using a different version of one of the scripts dependencies, I wondered if you knew of any tool that allowed the user to generate a list of all the AviSynth plugins (compiled DLLs, possibly also AVIS functions) installed and which version. I know in the StaxRip log it lists the names of installed plugins (but no further information). If there is nothing specific to AviSynth I can generate a sfv or md5 checksum of the cleanly installed plugins, if you are able to then run it and report back any different versions I might better able to isolate what exactly is causing the issue.

Thanks for all your help, really is appreciated.

Regards,

Ben

[StainlessS]

I wondered if you knew of any tool ...
This is as close as you will find.
http://forum.doom9.org/showthread.php?t=170647

[Desbreko]

Okay, when I get home I will try cleaning out the plugins (rename plugin directory, reinstall Avisynth v2.6). Can you tell me which avisynth.dll version you are using? I be barking up the wrong tree, but as of the lines that throw the error appear to be internal Avisynth commands it is possible...
I use Groucho's ICL build.

I tried to get the info tool, but Sendspace is derping and won't let me download it. I'll try again later.

[Groucho2004]

I tried to get the info tool, but Sendspace is derping and won't let me download it. I'll try again later.I added Dropbox mirrors to all my stuff.

[Desbreko]

Actually, I think the files themselves might be borked. Both the Sendspace and Dropbox downloads fail, yet I can download your other stuff from both sites with no problem.

[Groucho2004]

Actually, I think the files themselves might be borked. Both the Sendspace and Dropbox downloads fail, yet I can download your other stuff from both sites with no problem.
I don't know what to tell you - all that works for me.

[Desbreko]

Found the problem. Avast thought the exe was a virus and was cutting the download short.

Anyway, here's my plugin info for everything that edi_rpow2 uses: http://pastebin.com/hfqnPf29