AviSynthShader + SuperRes

[MysteryX]

Shiandow wrote a scaling algorithm called SuperRes that greatly enhances upscaling. Unfortunately, it wasn't possible to use this code in AviSynth and I really wanted to use it. So, I wrote an AviSynth plugin that allows running any HLSL pixel shaders through DirectX9.

Download the latest release here (v1.6.4, September 20th 2017)
https://github.com/mysteryx93/AviSynthShader/releases
Source code available on GitHub
https://github.com/mysteryx93/AviSynthShader

This plugin allows running HLSL pixel shaders within AviSynth. This gives access to various HLSL filters that haven't been programmed in AviSynth.

Note: Shiandow's SuperRes is not what is typically being called SuperRes; it's something else. It does not "create" any details like traditional SuperRes or Sharpening algorithms do. It doesn't have any temporal effect either, it works with frames one by one. Here's the way it works. It wraps around other resizers. After doubling the image size (with NNEDI3 for example), it resizes it back down with Bicubic and compares it with the original, producing a diff map representing details that were lost while upscaling. Then, it does its magic from that diff map. How? Well... with this code (https://github.com/zachsaw/MPDN_Extensions/blob/master/Extensions/RenderScripts/SuperRes/SuperResEx.hlsl). Results speak for themselves. I found it to work best with NNEDI3(nns=4)


Syntax information on GitHub (https://github.com/mysteryx93/AviSynthShader)


Special thanks to Shiandow for writing such amazing code, and especially to make it open source!

Special thanks to Madshi for taking the time to give very valuable pointers when nobody else was able to help!


Here are comparison images
ImageSource("Lighthouse.png").ConvertToRGB24()

Note: Since uploading these comparison, SuperRes has slightly changed. The newer version has less ringing and softer images.

1. Original
2. Spline16
3. nnedi3_rpow2(4, nns=4, cshift="Spline16Resize")
4. SuperXBR(edgeStrength=.6, weight=.6), twice
5. SuperRes(2, .43, 0, """edi_rpow2(2, nns=4, cshift="Spline16Resize")"""), twice
6. SuperResXBR(2, .6, xbrEdgeStrength=2.3, xbrSharpness=1.2), twice
7. SuperResXBR(1, .7, xbrEdgeStrength=.1, xbrSharpness=.7), twice

Lighthouse
http://s20.postimg.cc/kvxnnc315/Lighthouse.png (http://postimg.cc/image/kvxnnc315/) http://s20.postimg.cc/yqrcs282h/Lighthouse_Spline16.png (http://postimg.cc/image/yqrcs282h/) http://s20.postimg.cc/qzamti3x5/Lighthouse_NNEDI3.png (http://postimg.cc/image/qzamti3x5/) http://s20.postimg.cc/qjurmqsm1/Lighthouse_Super_XBR.png (http://postimg.cc/image/qjurmqsm1/) http://s20.postimg.cc/5jkzo0vyx/Lighthouse_Super_Res.png (http://postimg.cc/image/5jkzo0vyx/) http://s20.postimg.cc/rjx0o4s7t/Lighthouse_Super_Res_XBR.jpg (http://postimg.cc/image/rjx0o4s7t/) http://s20.postimg.cc/kag1j6315/Lighthouse_Super_Res_XBR_tweak.jpg (http://postimg.cc/image/kag1j6315/)

Clown
http://s20.postimg.cc/dmgoidduh/Clown.jpg (http://postimg.cc/image/dmgoidduh/) http://s20.postimg.cc/a44olzcyh/Clown_Spline16.png (http://postimg.cc/image/a44olzcyh/) http://s20.postimg.cc/5yts02f6h/Clown_NNEDI3.png (http://postimg.cc/image/5yts02f6h/) http://s20.postimg.cc/xjmlrbwpl/Clown_XBR.png (http://postimg.cc/image/xjmlrbwpl/) http://s20.postimg.cc/ze1xq1mft/Clown_Super_Res.png (http://postimg.cc/image/ze1xq1mft/) http://s20.postimg.cc/v290kit3t/Clown_Super_Res_XBR.jpg (http://postimg.cc/image/v290kit3t/) http://s20.postimg.cc/enjolp0ih/Clown_Super_Res_XBR_weak.jpg (http://postimg.cc/image/enjolp0ih/)

Eclipse (x2)
http://s20.postimg.cc/i3dmushg9/Eclipse.jpg (http://postimg.cc/image/i3dmushg9/) http://s20.postimg.cc/kpt5bq8nt/Eclipse_Spline16.png (http://postimg.cc/image/kpt5bq8nt/) http://s20.postimg.cc/c3pvr4wnt/Eclipse_NNEDI3.png (http://postimg.cc/image/c3pvr4wnt/) http://s20.postimg.cc/e0mlvpnbt/Eclipse_XBR.png (http://postimg.cc/image/e0mlvpnbt/) http://s20.postimg.cc/j4brn5brt/Eclipse_Super_Res.png (http://postimg.cc/image/j4brn5brt/) http://s20.postimg.cc/fjbknekt5/Eclipse_Super_Res_XBR.jpg (http://postimg.cc/image/fjbknekt5/)

[MysteryX]

SuperRes is one of may shaders that can be run with AviSynthShader. This particular shader greatly enhances upscaling quality and runs on top of any other algorithm you're already using such as NNEDI3.

SuperRes(input, passes, strength, softness, hqdownscaling, upscalecommand)

In Shaders\SuperRes\SuperRes.avsi. Thanks to Shiandow for writing this great code!

Arguments:

passes: How many SuperRes passes to run. Default=1.

strength: How agressively we want to run SuperRes, between 0 and 1. Default=1.

softness: How much smoothness we want to add, between 0 and 1. Default=0.

hqdownscaling: True to downscale using Bicubic, false to downscale using Bilinear.

upscalecommand: An upscaling command that must contain offset-correction. Ex: """nnedi3_rpow2(2, cshift="Spline16Resize")"""

Shiandow provides many other HLSL shaders available here that can be integrated into AviSynth.

https://github.com/zachsaw/MPDN_Extensions/tree/master/Extensions/RenderScripts

Here's a comparison of NNEDI3 with and without SuperRes.

Original
http://s30.postimg.cc/76lymfa3x/Original.jpg (http://postimg.cc/image/76lymfa3x/)

NNEDI3
http://s30.postimg.cc/53bjerab1/NNEDI3.jpg (http://postimg.cc/image/53bjerab1/)

NNEDI3 + SuperRes (passes=1, strength=1, softness=0)
http://s30.postimg.cc/b5j658yr1/Super_Res.jpg (http://postimg.cc/image/b5j658yr1/)

[MysteryX]

As of right now, ConvertFromFloat isn't working towards RGB32. By fixing this, it would allow avoiding an unnecessary RGB-YUV conversion within SuperRes and may increase quality and performance.

The conversion code is here, could someone familiar with RGB32 help me on this one?

https://github.com/mysteryx93/AviSynthShader/blob/master/Src/ConvertFromFloat.cpp#L54

Then, we could also avoid an unnecessary conversion by doing the SuperRes downscaling via a Shader; I don't have any such code right now, and I don't know if the DLL code will have to be tweaked to allow HLSL resizing.

Another limitation is that although the library contains a HLSL YUV-RGB conversion and converting back and forth works fine, I get weird distortion after running any other shaders. When using CPU conversion, it works fine. However, the CPU conversion code is Rec601 instead of Rec709. Since it gets converted back using the same algorithm, it doesn't distort the output, but it may cause a slight distort on the shader processing.

[foxyshadis]

You're officially my hero.

Edit: You're not incrementing dst properly. Since it's char, you're writing to it like:

00000000
ABC00000
ADEF0000
ADGHI000
etc.

A simple fix is swapping &dst[x],&dst[x+1],&dst[x+2] with &dst[(x*4)+2],&dst[(x*4)+1],&dst[x*4]. Note that I swapped the order because RGB32 is actually BGRA in memory.

The way the function's designed now isn't the greatest, but at least it'll work, that's most important.

[MysteryX]

Thanks. Now it almost looks OK but the image is still corrupted. It gives a similar effect as to when I'm trying to use the HLSL YUV-RGB conversion.

http://s8.postimg.org/kezny3pdt/RGB.jpg (http://postimg.org/image/kezny3pdt/)

I also added a "folder" parameter to SuperRes to specify where to find the .cso files.

[foxyshadis]

Is dst memsetted to 0? Could be garbage in the alpha bit. Actually, it's only happening on white, so it's probably not clamping. Should move the clamping outside of the if yuv/rgb block, along with memsetting (or just manually setting alpha to 0 each pixel).

[MysteryX]

Great, it now supports converting to/from RGB32. SuperRes's downscaling is now done in RGB32, avoiding unnecessary RGB-YUV conversion. It considerably further increases quality.

http://s1.postimg.org/ywdz41xsb/Super_Res_RGB.jpg (http://postimg.org/image/ywdz41xsb/)

2 areas that still need improvement:

1. Find a Bicubic HLSL shader to avoid unnecessary conversion for downscaling.

2. Convert YUV-RGB in Rec709 colorspace. Because it is currently converting in the Rec601 colorspace, the sharpening effect may slightly pull towards certain colors.

Contributions or suggestions are welcomed on those.

And then, of course, there are all kinds of performance optimizations that could be implemented.

Here are some performance numbers. CPU-Z isn't reading my GPU usage properly so I can't measure what's happening on the GPU.

nnedi3_rpow2(2, cshift="Spline16Resize")
60fps @ 55% CPU
SuperRes(1, 1, 0, true, """nnedi3_rpow2(2, cshift="Spline16Resize")""")
4.5fps @ 12% CPU

MT=8
nnedi3_rpow2(2, cshift="Spline16Resize", Threads=2)
80fps @ 83% CPU
SuperRes(1, 1, 0, true, """nnedi3_rpow2(2, cshift="Spline16Resize", Threads=2)""")
15fps @ 85% CPU

[Khanattila]

I have not had yet the opportunity to see the workings of Shiandow's SuperRes. Maybe is Farsiu or Mitzel. Really I have no idea.
I could make it a plugin like KNLMeansCL. If ever I have time.

[MysteryX]

SuperRes doesn't work like typical resize algorithms. It runs around other resizers. So let's say you want to use NNEDI3, it takes the larger image and the original image, does a Bicubic resize on the enlarged image to size it back down to the original, and then creates a difference map between the two, showing the details that were lost during the upsizing. From that diff map, it restores details and edges that were lost. Brilliant idea. It makes even basic resizers like Bilinear look decent.

Originally he was working on the Lab colorspace that definitely requires half-float data, and lately he dropped that to use Linear RGB colorspace. Perhaps I could try running with 8-bit-per-channel to see if there is a significant quality penalty.

[MysteryX]

I have added support for processing standard 8-bit-per-channel instead of 16-bit-per-channel. Simply add parameter "precision=1" to each conversion and shader calls. Quality is considerably lowered.

http://s17.postimg.org/hq0abhzob/Super_Res_Low_Precision.jpg (http://postimg.org/image/hq0abhzob/)

With MT=8
SuperRes(1, 1, 0, true, """nnedi3_rpow2(2, cshift="Spline16Resize", Threads=2)""")
I get 24fps @ 78% CPU

[Khanattila]

SuperRes doesn't work like typical resize algorithms. It runs around other resizers. So let's say you want to use NNEDI3, it takes the larger image and the original image, does a Bicubic resize on the enlarged image to size it back down to the original, and then creates a difference map between the two, showing the details that were lost during the upsizing. From that diff map, it restores details and edges that were lost. Brilliant idea. It makes even basic resizers like Bilinear look decent.

Originally he was working on the Lab colorspace that definitely requires half-float data, and lately he dropped that to use Linear RGB colorspace. Perhaps I could try running with 8-bit-per-channel to see if there is a significant quality penalty.
Fairsu: pdf (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCEQFjAAahUKEwiF6Le8mafIAhXEWxoKHVEpAqg&url=https%3A%2F%2Fusers.soe.ucsc.edu%2F~milanfar%2Fpublications%2Fjournal%2FSRfinal.pdf&usg=AFQjCNGJxos8M3zENBWM0hDpVBz2mpuFOw&sig2=13oKvRdCwKgFKLLcTueuiQ&cad=rja).
Mitzel: pdf (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB8QFjAAahUKEwjP0tPfmafIAhXHCBoKHSlnBSE&url=https%3A%2F%2Fvision.in.tum.de%2F_media%2Fspezial%2Fbib%2Fmitzel_et_al_dagm09.pdf&usg=AFQjCNGe47IRN26hgwFe3-VaNlS1smvEeQ&sig2=yZsH7IOp4OHMj8d8CmjJ_A&bvm=bv.104317490,d.d2s&cad=rja).

That's what I was talking about. Like me or tritical, we did not invent NLMeans, but we simply implement a well known algorithm :)

[MysteryX]

When using SuperRes, NNEDI3 and EEDI3 give almost identical output. Which means I can get rid of the ridiculously-expensive EEDI3.

1 pass with strength=.85 gives almost the same result as 2 passes with strength=.425, and 2 passes looks slightly better. 2 passes with NNEDI3 is faster than 1 pass with EEDI3. Splitting in 3 or more passes gives the exact same thing as 2 passes.

[MysteryX]

That's what I was talking about. Like me or tritical, we did not invent NLMeans, but we simply implement a well known algorithm :)
Shiandow isn't just implementing an existing algorithm, and this code isn't final. He's still working on it to try to find a better way to use Smoothness. Right now, it works better with Smoothness=0.

I'm seeing a weird bug when using 2 passes with strength=0.425, sometimes I get Access Violation, and it's random. Sometimes it starts anyway, and then fails unexpectedly. Now I'm running it with 1 pass with strength=0.85 and it's been running for half an hour without any issue. I really don't see why running a second pass could cause any such issue...

[vivan]

I have not had yet the opportunity to see the workings of Shiandow's SuperRes. Maybe is Farsiu or Mitzel. Really I have no idea.One thing that should be noted is that Shiandow's SuperRes is not a Super Resolution algorithm. I'd call it a "reverse downscaler" - it modifies upscaled image so that downscaled upscaled image is closer to the original image.

[Shiandow]

One thing that should be noted is that Shiandow's SuperRes is not a Super Resolution algorithm. I'd call it a "reverse downscaler" - it modifies upscaled image so that downscaled upscaled image is closer to the original image.

It's called "single frame super resolution" in literature though. The current implementation is most closely related to the method described in this paper (http://vision.eecs.northwestern.edu/papers/conference/2007/ICME07_Dai_final.pdf). The algorithms described by Faisru and Mitzel are for multi frame super resolution, which uses similar techniques but with a different goal.

[MysteryX]

I fixed a crash when the DX9 device was lost.

I also replaced the code to copy the data in and out of DX9 with AviSynth's BitBlt, and it considerably increases performance by itself.

input.ConvertToFloat().ConvertFromFloat() renders 38fps @ 12% CPU, and if I disable YUV-RGB conversion, 42fps. Could someone look at how this short code could be optimized?

[MysteryX]

The Access Violation error seems to be an out of memory error, plain and simple.

After playing some more with the settings, softness is actually working pretty well when using Strength=1 and 2 or 3 passes.

This is the best result I got with Passes=3, Strength=1, Softness=.25

http://s9.postimg.org/msn9rajyj/Super_Res3pass.jpg (http://postimg.org/image/msn9rajyj/)

There should be further increase in quality once I get to do Bicubic downscaling via shader, as it will avoid clamping on the diff map.

[MysteryX]

One thing I love about SuperRes is that before, to get the best quality upscaling, I would have to combine EEDI3 and NNEDI3 in the right order and add just the right amount of sharpening at 3 different stages.

With SuperRes, I get better quality with only NNEDI3 and no artificial sharpening. No fine-tuning is necessary per video, standard NNEDI3 frame doubling with SuperRes(Passes=2, Strength=1, Softness=.3) is working perfect for all the videos I tried so far.

For the media encoder I was working on, it will make it much simpler.

[MysteryX]

Updated the code and DLL.

- Increased performance
- Added Shader Width and Height parameters to set the output frame size. Default = same as source clip.

[MysteryX]

Fixed float-byte rounding to be more accurate by adding .5f before rounding. Slight performance improvement.

This results in the colors being slightly brighter, and the SuperRes Diff map to be more accurate which slightly improve its effectiveness.

I could make it a plugin like KNLMeansCL. If ever I have time.
As far as writing a native AviSynth version, I don't know if that would work. Originally, Shiandow was using the Lab colorspace which definitely requires half-float processing. He finally dropped it to use RGB Linear (not Gamma) colorspace. I doubt the YUV-RGB conversion could be avoided, and from my tests processing it with non-float data, the quality is considerably lower. This algorithm is very sensitive to details and must be processed with half-float precision. In that sense, perhaps native approaches wouldn't even be better than this. The GPU is much better at processing float data than the CPU.

[Ghostlamer]

Is there any way to make it work with rfactor larger than 2.
My script:

XviD4PSPPluginsPath = "C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\"

LoadPlugin(XviD4PSPPluginsPath+"nnedi3.dll")
LoadPlugin(XviD4PSPPluginsPath+"Shader.dll")

Import(XviD4PSPPluginsPath+"SuperRes.avsi")

SetMTMode(3,3)


AviSource("F:\2K\00.avi", audio=false, pixel_type="YV12")



SetMTMode(2)

SuperRes(2, 1, 0, false, """nnedi3_rpow2(rfactor=4, cshift="Spline16Resize", Threads=1)""")

Spline64Resize(2560,1380)

Without mtmode and rfactor 4 - stable working, but with 0.80fps with 6% cpu usage and 0% gpu, if i use rfactor=4 and setmtmode with more than 2 threads - crash, with 2 threads 1.40-1.60 fps and very low cpu and gpu usage.
Resolurion of original video 712x384.

[MysteryX]

It crashes because you're going past the 2GB memory limit. The code will need to be optimized.

Performance can be improved by rewriting the functions to convert frames to/from float. The float/halffloat conversion can be done with a buffer instead of 1 by 1 which probably would increase performance. Having a HLSL Bicubic resize function also would help.

As far as memory usage, I'm not sure what can be done. Each DirectX 9 device is creating its own threads and managing its own memory. A DX9 device is created each time a Shader is called. If there are 8 shader calls within SuperRes, and 4 threads, then that's 32 DX9 devices.

You can analyze your script with AVSMeter. Using 1 pass instead of 2 also will increase performance. According to my tests, NNEDI3 works best with 2 threads.

Try this
SuperRes(1, .85, 0, false, """nnedi3_rpow2(rfactor=4, cshift="Spline16Resize", Threads=2)""")

[Khanattila]

Fixed float-byte rounding to be more accurate by adding .5f before rounding. Slight performance improvement.

This results in the colors being slightly brighter, and the SuperRes Diff map to be more accurate which slightly improve its effectiveness.


As far as writing a native AviSynth version, I don't know if that would work. Originally, Shiandow was using the Lab colorspace which definitely requires half-float processing. He finally dropped it to use RGB Linear (not Gamma) colorspace. I doubt the YUV-RGB conversion could be avoided, and from my tests processing it with non-float data, the quality is considerably lower. This algorithm is very sensitive to details and must be processed with half-float precision. In that sense, perhaps native approaches wouldn't even be better than this. The GPU is much better at processing float data than the CPU.

Have you tried to normalize data in uint32_t? Without use floating-point numbers.
If working with processor is much faster.
unorm32 = (UINT32_MAX * (value - VALUE_MIN)) / (VALUE_MAX - VALUE_MIN)

If for example VALUE_MIN is 0 and VALUE_MAX is 255.
unorm32 = (4294967295 * value) / 255 = 16843009 * value
0 --> 0
1 --> 16843009
2 --> 33686018
...
255 --> UINT32_MAX

[Groucho2004]

Have you tried to normalize data in uint32_t? Without use floating-point numbers.
If working with processor is much faster.
Indeed. What's up with the obsession of some folks using floats lately? Even 64 bit (u)int is faster than 32 bit float.

[MysteryX]

Have you tried to normalize data in uint32_t? Without use floating-point numbers.
I could do some more experiment with that, the shader processing can be done with uint data. However, the way color conversion is currently done creates overflow so that won't work just yet; there would be "stuff" to fix in the color conversion first. The HLSL color conversion code should avoid overflow, but I was seeing weird behaviors when I tried using it so that's not yet working.

If we can get color conversion to avoid overflow and work properly, then we could try processing with uint data and see what performance difference it makes.

[MysteryX]

I'm leaving to China for 2 weeks and won't be playing with this. If someone wants to look into the code, you could look into
1. Getting the HLSL color conversion to work; or having CPU conversion that avoids overflows
2. Getting the shader to run with uint data (changing buffer format); which requires not having overflows
3. Optimizing the ConvertToFloat and ConvertFromFloat functions. Precision could have 3 values for ConvertToFloat, ConvertFromFloat and Shader functions: 1 (8-bit per channel), 2 (16-bit uint per channel) or 3 (16-bit float per channel)

[luigizaninoni]

I must be missing something obvious, but have you any idea why is AvsPMod giving error:

LoadPlugin: unable to load "c:\users\admin\desktop\shaders\shader.dll",Module not found. Install missing library ?

shader.dll is definitely in that directory

My script:
LoadPlugin("C:\Users\admin\Desktop\Video\Staxrip\Applications\DGMPGDec\DGDecode.dll")
LoadPlugin("c:\users\admin\desktop\shaders\shader.dll")
MPEG2Source("C:\Users\admin\Desktop\10-07-05-40-01-mozzibrb-TELECOLOR temp files\10-07-05-40-01-mozzibrb-TELECOLOR.d2v",cpu=6,ipp=true,moderate_h=40,moderate_v=60,idct=5)
Crop(2, 2, -2, -2)
QTGMC(Preset="Slow")
SelectEven()
SuperRes(2, 0.85, 0, true, """nnedi3_rpow2(rfactor=2, cshift="Spline16Resize", Threads=2)""", "C:\users\admin\desktop\shaders\")

[Ghostlamer]

It crashes because you're going past the 2GB memory limit. The code will need to be optimized.

Im using virtualdub and when the crash occurs, vdub process eat only 800-1100 mb.
Script working with setmtmode(3 and 5) instead of 2 (with more than 2 threads), but the output video - buggy (mode 3, 4-5 fps), low perfomance (mode 5, 0.5-0.8 fps).

Try this
SuperRes(1, .85, 0, false, """nnedi3_rpow2(rfactor=4, cshift="Spline16Resize", Threads=2)""")
Thanks for the advice.

[MysteryX]

btw, if anyone wants to play with the code, it's pretty simple, but you need
- DirectX SDK (http://www.microsoft.com/en-us/download/details.aspx?id=6812)
- Visual Studio (I'm making very little use of C++, it could be adapted to standard C with little changes)

GitHub allows you to download the source code, make your own changes and upload your contributions to the code. It takes some time to learn how use but then is very useful for collaborative projects. TortoiseGit makes it much easier to use.

[Groucho2004]

I must be missing something obvious, but have you any idea why is AvsPMod giving error:

LoadPlugin: unable to load "c:\users\admin\desktop\shaders\shader.dll",Module not found. Install missing library ?

shader.dll is definitely in that directory
Which OS are you using? If you're using Vista or above, use Dependency Walker to find out what's missing, possibly some of the DX stuff.

[Ghostlamer]

MysteryX, Triple quotes can be bypassed?, why I ask?, there is a script mt_pipeline http://forum.doom9.org/showthread.php?t=163281 , it allows to bypass the 2gb limit, but it also uses triple quotation marks and conflicts with supereres (but works very well with many others), i just not avisynth guru, do not know much.

[MysteryX]

I tried applying the 4GB patch to AVSMeter.exe, and it didn't work. The flag D3DXCONSTTABLE_LARGEADDRESSAWARE must also be added within the DX9 source code to make that work. Not sure why it hasn't worked yet.

[Groucho2004]

I tried applying the 4GB patch to AVSMeter.exe, and it didn't work.
LARGEADDRESSAWARE is one of linker options I use for the 32 Bit binary. No need to patch.

[luigizaninoni]

Which OS are you using? If you're using Vista or above, use Dependency Walker to find out what's missing, possibly some of the DX stuff.

Problem solved. d3dx9_43.dll was actually missing. Thank you very much for your kind advice

[MysteryX]

Problem solved. d3dx9_43.dll was actually missing. Thank you very much for your kind advice
How come was that file missing? Isn't it a system file that "should" already be there?


I just did an experiment with processing frames with int data instead of float. When initializing the device, I replaced the format D3DFMT_A16B16G16R16F by D3DFMT_A16B16G16R16. The performance is slightly faster but not that much. We might save some more in the data conversion. Obviously, with this test, the image was corrupt because it was processing half-float data as if it was uint, but it looked better than I would have expected.

There is definitely a bottleneck somewhere and it isn't the half-float shader processing.

In terms of numbers, I'm using this script
SetMTMode(3,4)
AviSource("Preview.avi", audio=false, pixel_type="YV12")
SetMTMode(2)
SuperRes(2, .42, 0, true, """nnedi3_rpow2(2, cshift="Spline16Resize", Threads=2)""")
Distributor()

With D3DFMT_A16B16G16R16 or D3DFMT_A16B16G16R16F, I get almost exactly the same numbers: 12fps @ 53% CPU. Memory usage also is the same.

One thing I found out is that creating a .def file (such as AVSMeter.def) with "STACKSIZE 512KB" in it slightly increases performance.

I also just did another quick test: removing the half-float conversions. It still calculates in float but then converts into short. Performance went way up from 12fps to 15-16fps.

[Shiandow]

Have you tried to normalize data in uint32_t? Without use floating-point numbers.
If working with processor is much faster.
unorm32 = (UINT32_MAX * (value - VALUE_MIN)) / (VALUE_MAX - VALUE_MIN)

If for example VALUE_MIN is 0 and VALUE_MAX is 255.
unorm32 = (4294967295 * value) / 255 = 16843009 * value
0 --> 0
1 --> 16843009
2 --> 33686018
...
255 --> UINT32_MAX

Indeed. What's up with the obsession of some folks using floats lately? Even 64 bit (u)int is faster than 32 bit float.

Well ,the original SuperRes code (designed for MPDN) used 16 bit uint for most of the processing. It does store an intermediate results in float, but that conversion is handled by the GPU itself. I'm not even sure if that part is necessary, signed ints would probably work just as well.

However the shaders will still use floats (single precision) internally. And as far as I know GPUs aren't that good at integer (or fixed point) arithmetic, but maybe that's changed.

[Khanattila]

Well ,the original SuperRes code (designed for MPDN) used 16 bit uint for most of the processing. It does store an intermediate results in float, but that conversion is handled by the GPU itself. I'm not even sure if that part is necessary, signed ints would probably work just as well.

However the shaders will still use floats (single precision) internally. And as far as I know GPUs aren't that good at integer (or fixed point) arithmetic, but maybe that's changed.

GPU are TERRIBLE with integer. But it have a fast internal conversion from integer to float.

Like KNLMeansCL, this is the way forward:
Read Integer Buffer --> GPU internal Conversion to normalized float --> Processing float --> GPU internal Conversion to integer --> Write to Integer Buffer.

Anyway, in this case it is better not to use float rather than converting by CPU.

[MysteryX]

I wouldn't be surprised if the DX9 function to convert half-float data is delegated to the GPU then, and that's what the buffer-processing function is for. If that's the case, then right now I'm sending commands to the GPU one by one. If I batch them into a buffer to be processed all at once, then performance would probably be MUCH better. Worth a try!

[MysteryX]

I edited ConvertToFloat to use a buffer for half-float conversions. It still calculates as float (which could be optimized by calculating int instead), stores all data into a large float buffer, converts the whole frame at once, then copy back into the frame. ConvertFromFloat doesn't have those changes yet.

That change brought the performance up from 12fps to 14.5fps.

[MysteryX]

ConvertToFloat and ConvertFromFloat are now both using a buffer for half-float conversion. Performance is now 18.5fps instead of 12fps. It could be further improved by calculating int data instead of float.

With this optimization, the CPU usage is also now higher, even with only 4 threads, so the whole script is running considerably faster.

Edit: I adapted ConvertToFloat to calculate color conversion with INT instead of FLOAT, performance further increased.

[MysteryX]

I further increased performance by using DirectXMath DirectX::PackedVector::XMConvertFloatToHalfStream instead of D3DXFloat32To16Array.

It went from 18.5fps to 20fps. CPU usage at only 40%.

ConvertToFloat is faster when calculating in INT, but ConvertFromFloat is faster with FLOAT than with INT.

[luquinhas0021]

SuperRes doesn't work like typical resize algorithms. It runs
around other resizers. So let's say you want to use NNEDI3, it
takes the larger image and the original image, does a Bicubic resize
on the enlarged image to size it back down to the original, and
then creates a difference map between the two, showing the
details that were lost during the upsizing. From that diff map, it
restores details and edges that were lost. Brilliant idea. It makes
even basic resizers like Bilinear look decent. Is not I`m thinking your work is bad, but it looks like to me more like a detail restoration than detail "add", case of super resolution. In single image, sr works by searching seems patterns and slightly different details in each one of them.

[MysteryX]

New version v0.9.1 is released. It greatly reduces memory usage!! This version allows running several shaders in a row by creating command chains and calling ExecuteShader() at the end.

https://github.com/mysteryx93/AviSynthShader

As for SuperRes, it cannot yet fully benefit from this as I'm still missing a Bicubic downscaling shader that needs to be run in the middle. I can only combine 2 of the shader calls (twice if doing 2 passes), yet that's enough to considerably reduce memory usage. ConvertToFloat and ConvertFromFloat have also been modified to reduce memory usage.

With this version, you'll be able to run 8 threads without any issue.

If I can get a Bicubic downscaler, then we could remove unnecessary ConvertFromFloat and ConvertToFloat, as well as chain all of the commands to run at once, which would greatly improve memory usage and performance.

This Cubic code would work for Bicubic upscaling, but Bicubic downscaling requires a few tweaks. I can't do this as I know nothing about HLSL programming.
https://github.com/zachsaw/MPDN_Extensions/blob/master/Extensions/RenderScripts/ChromaScaler/Chroma.hlsl

[MysteryX]

Here's a comparison of the image quality.

Original
Spline16
NNEDI3(nns=4)
NNEDI3(nns=4)+SuperRes(passes=2, strength=.42)
http://s21.postimg.org/na30zwo2r/Original.png (http://postimg.org/image/na30zwo2r/)http://s21.postimg.org/vtmexnwf7/Spline16.png (http://postimg.org/image/vtmexnwf7/)http://s21.postimg.org/71msqff8j/NNEDI3.png (http://postimg.org/image/71msqff8j/)http://s21.postimg.org/amio9njs3/NNEDI3_Super_Res.png (http://postimg.org/image/amio9njs3/)

Result speak for themselves. It makes the image shaper without creating any artificial details.

If I eventually get a Bicubic HLSL downscaling, there might be a 'slight' further quality improvement.

[wonkey_monkey]

Here's a comparison of the image quality.

Result speak for themselves.

I find it very hard to see any difference. I don't think your choice of test image was a very good one - it doesn't have a lot of detail and looks very JPEGy.

[luquinhas0021]

MysteryX, I didn't speak any SR algorithm creates artificial details. What I wanted say is common SR algorithms analizes a set of neighbour frames or, in a single image, a lot of similar patterns, in order to, let's say, replicate details from a frame to another.
Your algorithm first upscale image using some algorithm, then downscale it using BiCubic and compare it to original image, creates a difference map, upscale the missing parts and paste into upscale image. This was what I understood.
But, generally, upscaling are detail lossy and downscaling, too. So, probably, the difference map will show so much difference. What I really want know is how the details that are in original image, but aren't in downscaled upscaled image, are pasted into upscaled image, i.e, how this details are upscaled.
David Horman, I don't see so much difference too. The maximum I see was some ringing, in SuperRes image, disappear.

[Bloax]

https://dl.dropboxusercontent.com/u/63152810/itnse3.png
Here's a nice still image if you need one. :-)

[MysteryX]

Here's some tests with the lighthouse and clown.

Original
Spline16
NNEDI3(nns=4, cshift="Spline16Resize")
NNEDI3+SuperRes(passes=2, strength=.42)
SuperRes+nnedi3_rpow2(rfactor=2, nsize=0, nns=4, qual=2, etype=0, cshift="SincResize", ep0=4, threads=0, opt=0, fapprox=0)
NNEDI3(nns=4, cshift="Spline16Resize")+SuperRes(passes=3, Strength=1, Softness=.85)

http://s24.postimg.org/qpoo33g41/Lighthouse_Original.png (http://postimg.org/image/qpoo33g41/)http://s24.postimg.org/o9mup8y1d/Lighthouse_Spline16.png (http://postimg.org/image/o9mup8y1d/)http://s24.postimg.org/fotiy2nv5/Lighthouse_NNEDI3.png (http://postimg.org/image/fotiy2nv5/)http://s24.postimg.org/3qryk6k41/Lighthouse_Super_Res.png (http://postimg.org/image/3qryk6k41/)http://s24.postimg.org/ep3olqpn5/Lighthouse_Sinc.png (http://postimg.org/image/ep3olqpn5/)http://s9.postimg.org/7cbmxgi0r/Lighthouse_Soft.jpg (http://postimg.org/image/7cbmxgi0r/)


http://s24.postimg.org/k85rjl5qp/Clown_Original.png (http://postimg.org/image/k85rjl5qp/)http://s24.postimg.org/qx7plv1ox/Clown_Spline16.png (http://postimg.org/image/qx7plv1ox/)http://s24.postimg.org/5ma5hljkh/Clown_NNEDI3.png (http://postimg.org/image/5ma5hljkh/)http://s24.postimg.org/n3iuqg9qp/Clown_Super_Res.png (http://postimg.org/image/n3iuqg9qp/)http://s24.postimg.org/panjxqvyp/Clown_Sinc.png (http://postimg.org/image/panjxqvyp/)http://s7.postimg.org/ewqjjvsxj/Clown_Soft.png (http://postimg.org/image/ewqjjvsxj/)


I see more difference between NNEDI3 and NNEDI3+SuperRes than between Spline16 and NNEDI3. There's also something funny happening with the reds... the reds are different but actually looks better with SuperRes. I've seen in a video with a chair where half of it was plain red (color cropping), that the texture of the chair somehow came back after passing it through SuperRes and it looked more like a chair afterwards. It must have to do with the way it's doing color conversion, but it's accidental. It seems to 'sometimes' recover cropped colors. Somehow. Another time I've seen it turn overflow colors into some other color

[MysteryX]

luquinhas0021, I couldn't answer the technical of how it's doing its job internally, but in SuperRes.avsi, you can see the diff map by returning the output of SuperResDiff.cso instead of processing the image with it.

Bloax, here's the result with your image

http://s10.postimg.org/7omfwhbcl/Eclipse_Original.png (http://postimg.org/image/7omfwhbcl/)http://s10.postimg.org/bz13s2gfp/Eclipse_Spline16.png (http://postimg.org/image/bz13s2gfp/)http://s10.postimg.org/madgkq851/Eclipse_NNEDI3.png (http://postimg.org/image/madgkq851/)http://s10.postimg.org/amjeq6j05/Eclipse_Super_Res.png (http://postimg.org/image/amjeq6j05/)http://s24.postimg.org/cyknk9841/Eclipse_Sinc.png (http://postimg.org/image/cyknk9841/)http://s7.postimg.org/wpc3rra5z/Eclipse_Soft.png (http://postimg.org/image/wpc3rra5z/)

[luquinhas0021]

Nice, MysteryX. The differences from nnedi3 upscaling to superres nnedi3 upscaling, at least the ones I realize, are less haloing and more sharpness. What if use the Sinc4, Lanczos4 and apply super resolution in each one?! I believe you apply SR in nnedi3 nns=4 with default parameters. What would happen if SR was applied in this script...?

nnedi3_rpow2(rfactor=2, nsize=0, nns=4, qual=2, etype=0, cshift="SincResize", ep0=4, threads=0, opt=0, fapprox=0)