Avisynth AiUpscale v1.2.0

[Alexkral]

This is a script to test some Super-resolution convolutional neural networks implemented as hlsl pixel shaders, through Avisynth Shader.

Download (https://github.com/Alexkral/AviSynthAiUpscale/releases/tag/1.2.0)

Github repository with explanation on its usage:

https://github.com/Alexkral/AviSynthAiUpscale

I became interested in this after reading some posts in which it was assumed that implementing the FSRCNNX shaders using AviSynth Shader was not possible, because they were written in GLSL. After all, GLSL and HLSL are not that different, and the mpv hook format doesn't change this. So I was able to do it without too much trouble, also with the other FSRCNN shaders by igv and haasn. That is what I thought to share at the time, but since igv provides the code not only to train the networks, but also to convert the weights into mpv hooks, I became interested in training different models. This also I think has worked very well (quite boring I must say, except for the part related to experimentation), and I could have shared the result but, how not, I became interested in adding other kinds of networks, which also allowed me to do other kinds of experiments. Finally, I realized that using a specific dataset for training allowed to obtain better results for specific content, which made me repeat the entire training process for the "Illustration" models.

So it's been a while since I started with this, every single part has needed to be redone multiple times, lots of training and retraining, having to deal with things that I didn't even know existed, such as the chroma positioning when resizing or some AviSynth Shader specific problems, so I can't be happier to have finished it at last.

Special thanks to MysteryX, igv and Shiandow for writing such amazing code without which this couldn't be possible.

Here are some results on performance tested with AVSMeter for a YV12 clip with default settings (chroma upscaled with Lanczos). GPU is a Nvidia GTX 1080:

1080p -> 2160p

AiUpscale Fast = 38.47 fps
AiUpscale Medium = 25.30 fps
AiUpscale HQ = 17.16 fps

[real.finder]

thanks!

I did try it and it work with server that don't has gpu, is this ok? but anyway it a good thing :)

edit: FFT3DGPU also work! seems directx things can work with cpu only

[goorawin]

Your AiUpscale looks very interesting, so I'm interested to test it against other upscaling options.
You mention that it is able to adjust the target width and height.
The 4x Upscaling looks good.
If that is the case could you please provide a sample script that upscales from 720x576SD to 1920x1080HD and HD to UHD.

[Alexkral]

If that is the case could you please provide a sample script that upscales from 720x576SD to 1920x1080HD and HD to UHD.

AiUpscale (clip, Luma = "VDSR", Width = 1920, Height = 1080)

This will scale the 720x576 clip to 1920x1080. Since the height scaling ratio is 1.875, only the VDSR model can be used. When the ratio is at least 2 (and less than or equal to 4) for both width and height, the HQ and HQ Sharp models can also be used.

For HD to UHD it is the same.

[poisondeathray]

Thanks for this. I'm having problems getting it working


Shader: Failed to open pixel shader Shaders\Photo\Medium\2x\feature_1.cso
AiUpscale.avsi, line 393
line 217
line 218
line 239


I think it's some path related issue. Win8

AiUpscale.avsi and Shaders folder are in the avisynth+ plugins directory. AviSynthShader works ok for me separately .

(The .avsi says "AviSinth+ plugins folder", but I'm assuming it's obviously a typo, the github page says "AviSynth+ plugins directory", but maybe you can edit it for the next release)

Does mpv need to be installed ? I installed it anyways but still same error; I normally use a "portable" version, but I might uninstall if you technically don't need it. I have mpv.conf in AppData\Roaming\mpv and edited some entries .

profile = gpu-hq
glsl-shader = "<PATH TO HOOK>"

Where the actual path of mpv.hooks is located; and I tried different things such as specifying the directory above (container mpv.hooks folder), and "/" vs. "" slashes , I tried escaping colon, tried ' instead of " , I tried specifying specific .glsl files in the mpv.hooks directory

[Alexkral]

That's weird, the path seems correct.

AiUpscale.avsi and the Shaders folder should be in the same folder as AviSynth Shader, on my PC that is the AviSynth+\plugins64+ folder.

I have only tested it on Win7 and Win10 x64, but I don't see any reason why it won't work with x86.

Other than that you could try opening VirtualDub2 with administrator privileges to see if that makes any difference, and then open the script.

mpv is only needed to use the hooks, if you extract them in the same folder as mpv.conf, you would have to edit it for example like this:

profile = gpu-hq
glsl-shader = "~~/AiUpscale_Fast_2x_Photo.glsl"

[real.finder]

I think the script need to use ScriptDir() http://avisynth.nl/index.php/Internal_functions#ScriptDir

[poisondeathray]

I have only tested it on Win7 and Win10 x64, but I don't see any reason why it won't work with x86.


Sorry it is x64

My plugins are in a custom directory location, not default (registry entries are ok , everything else works, other scripts/plugins). Could that be related?



Other than that you could try opening VirtualDub2 with administrator privileges to see if that makes any difference, and then open the script.


no difference


mpv is only needed to use the hooks, if you extract them in the same folder as mpv.conf, you would have to edit it for example like this:

profile = gpu-hq
glsl-shader = "~~/AiUpscale_Fast_2x_Photo.glsl"


I tried moving them and this configuration too, same error

[Alexkral]

Just to be clear, the mpv hooks are not required at all for AviSynth AiUpscale to work, and the path in mpv.conf is also not used. They are included only to be used with mpv.

Please change line 109 in AiUpscale.avsi to this and report if this fixes the problem:

Global Path = ".\Shaders\"

[poisondeathray]

Just to be clear, the mpv hooks are not required at all for AviSynth AiUpscale to work, and the path in mpv.conf is also not used. They are included only to be used with mpv.

Got it, thanks.



Please change line 109 in AiUpscale.avsi to this and report if this fixes the problem:

Global Path = ".\Shaders\"

No , similar error

Shader: Failed to open pixel shader .\Shaders\Photo\Medium\2x\feature_1.cso
(rest of message is the same)

[Alexkral]

Sorry but I can't figure out how to fix this since AviSynth doesn't provide anything to find out the path to the plugins directory. It should work if you specify the full path to the Shaders folder (ended with a backslash and in quotes) on line 109, but obviously that's not going to help others with the same problem.

[poisondeathray]

Sorry but I can't figure out how to fix this since AviSynth doesn't provide anything to find out the path to the plugins directory. It should work if you specify the full path to the Shaders folder (ended with a backslash and in quotes) on line 109, but obviously that's not going to help others with the same problem.

Yes, this works for me. Was it just me having the problem ?

Global Path = "<full path...\Shaders\>"


I don't understand why either, because I'm pretty sure other working scripts use that variable method too.

Thanks again

[real.finder]

Sorry but I can't figure out how to fix this since AviSynth doesn't provide anything to find out the path to the plugins directory. It should work if you specify the full path to the Shaders folder (ended with a backslash and in quotes) on line 109, but obviously that's not going to help others with the same problem.

as I said here https://forum.doom9.org/showthread.php?p=1919111#post1919111 anyway see my Pull request

[Alexkral]

as I said here https://forum.doom9.org/showthread.php?p=1919111#post1919111 anyway see my Pull request

Thanks but this doesn't work since ScriptDir() returns the path of the loaded script, not the .avsi

[real.finder]

Thanks but this doesn't work since ScriptDir() returns the path of the loaded script, not the .avsi

you sure? it's work fine here from the avsi

[real.finder]

poisondeathray, can you try it? https://github.com/realfinder/AviSynthAiUpscale/raw/master/AiUpscale.avsi

[poisondeathray]

poisondeathray, can you try it? https://github.com/realfinder/AviSynthAiUpscale/raw/master/AiUpscale.avsi

Different error message. The current directory location of the .avs script is location is appended (but Shaders directory actually isn't there)

Shader: Failed to open pixel shader F:\test\Shaders\Photo\Medium\2x\feature_1.cso

If I copy/paste Shaders folder to that F:\test folder, it works

[real.finder]

Different error message. The current directory location of the .avs script is location is appended (but Shaders directory actually isn't there)

Shader: Failed to open pixel shader F:\test\Shaders\Photo\Medium\2x\feature_1.cso

If I copy/paste Shaders folder to that F:\test folder, it works

but it work fine for me, maybe because I made it load from subfolder by another avsi

anyway

make an avsi named AviSynthPluginsDir.avsi and put this in it


global AviSynthPluginsDir = VersionNumber() < 2.60 ? RT_GetWorkingDir() : ScriptDir()


and put it in your plugins folder

and replace ScriptDir() with AviSynthPluginsDir in AiUpscale.avsi

[poisondeathray]

r.f - it works.

I deleted the copied shaders folder, find/replaced all 4 entries in AiUpscale.avsi as per instructions

Thanks to you both

[Alexkral]

Some tests here:

https://forum.doom9.org/showthread.php?p=1919165#post1919165

[DTL]

Strange to see Lanczos resize as reference for video data. I think the SincResize (with max possible 'taps' parameter) is closer to reference upsize resampler for band-limited and 'conditioned' against ringing good quality video data. Lanczos is more universal, but it significally limited for reference restoring of highset frequencies in video data. Unfortunately current version avisynth looks like has (small) bug in SincResize resampler implementation and it can make compare metrics with ideal input reference image data worse.

[Alexkral]

AviSynth AiUpscale v1.0.1 (https://github.com/Alexkral/AviSynthAiUpscale/releases/tag/v1.0.1)

Changes:

- Added shaders source code.
- Some minor fixes.

[wolfman2791]

This is very good. Thank you!

[gmail123]

how to import to vs?

[Alexkral]

This is very good. Thank you!
You're welcome. :)
how to import to vs?
I'm pretty new to VapourSynth so I can't help you with that, sorry.

[poisondeathray]

how to import to vs?

you can run glsl shaders directly in vs with vs-placebo

https://github.com/Lypheo/vs-placebo


If you're on Windows with avisynth installed, you can also import an avs script into vapoursynth with AVISource

[gmail123]

you can run glsl shaders directly in vs with vs-placebo

https://github.com/Lypheo/vs-placebo


If you're on Windows with avisynth installed, you can also import an avs script into vapoursynth with AVISource

my english is poor...
sample:1080p ---->2160P,
can you give me a vs script ?
thx！

[poisondeathray]

my english is poor...
sample:1080p ---->2160P,
can you give me a vs script ?
thx！


My English is poor too (and it's my first language :D)


You should probably post in the vapoursynth subforum

One difference is Alexral did some separate training (slightly different) . I'm not sure how to use those into vapoursynth directly; only .glsl shaders can be used with placebo.Shader . (You can use the AVISource workaround, but you need windows, avisynth installed)

There is documentation on the page


eg.


import vapoursynth as vs
core = vs.get_core()
clip = core.lsmas.LWLibavSource(r'PATH\input.mkv')
clip = core.resize.Bicubic(clip, format=vs.YUV444P16)
clip = core.placebo.Shader(clip, width=clip.width*2, height=clip.height*2, shader=r'PATH\FSRCNN-Medium\FSRCNN_x2_r2_16-0-2.glsl')
clip.set_output()

[Alexkral]

Yes, there are no glsl mpv hooks of the VDSR and HQ models because they were fully or partially trained with Matlab, and their weights were converted directly to hlsl. They would have been too slow for mpv anyway.

[biship]

This can be used in MadVR?

[Alexkral]

This can be used in MadVR?

No, they are different programs.

[gmail123]

compare to with Waifu2x- cunet (For 2D artwork. Slow, but better quality.)?

[Alexkral]

compare to with Waifu2x- cunet (For 2D artwork. Slow, but better quality.)?

That's subjective. For RGB it seems that the VDSR and HQ models are a little slower. Looking at the results, I think you'll find that the AiUpscale models are a bit sharper.

[gmail123]

That's subjective. For RGB it seems that the VDSR and HQ models are a little slower. Looking at the results, I think you'll find that the AiUpscale models are a bit sharper.

I had tested it,VDSR/HQ models are very good.

1、Plan to support 32bit float?
2、for video, "Photo" or "Illustration"，Which is better?

[Alexkral]

32 bit float is not supported because the processing in Avisynth Shader is done at 16 bit depth, I don't think that will change.

For video that comes from a camera you should use the "Photo" models, the "Illustration" models are for artwork, drawings, etc.

[gmail123]

32 bit float is not supported because the processing in Avisynth Shader is done at 16 bit depth, I don't think that will change.

For video that comes from a camera you should use the "Photo" models, the "Illustration" models are for artwork, drawings, etc.

good,thx!

[markiemarcus]

This is really fantastic work. Thank you so much for all of the effort! These NN upscalers can be a little unpredictable and being able to modify the script and check the results in realtime is just invaluable. Really, really, terrific work. It's incredibly fast, too. Absolutely flies on my RX 480.

[markiemarcus]

Quick question. How is this handling 601 to 709 conversion, if at all? The SD sources I'm working from are all 601 so there's a huge shift post upscale. Is there an optimal way of handling this?

[Alexkral]

Quick question. How is this handling 601 to 709 conversion, if at all? The SD sources I'm working from are all 601 so there's a huge shift post upscale. Is there an optimal way of handling this?

Hi, thanks for the feedback. There is no conversion between RGB and YUV so there should be no change in this. If you're using VirtualDub2 you have to select that in Video > Decode format.

[markiemarcus]

Hi, thanks for the feedback. There is no conversion between RGB and YUV so there should be no change in this. If you're using VirtualDub2 you have to select that in Video > Decode format.

Spot on, thanks for getting back to me. Again, this is just fab work and much appreciated.

[gmail123]

hq/vdsr model，a lot of Mosquito noise,how to get rid of?...

[hungryeyes]

thank goodness for this script. I've been trying to get waifu2x to work without having to write temporary files (am noob) and this looks like it will work way better. I'm particularly anxious to see how it will handle old DVDs with interlacing.

[Alexkral]

I'm particularly anxious to see how it will handle old DVDs with interlacing.

Interlaced content is not supported, you'll have to deinterlace first.

[hungryeyes]

some of the first ai upscalers started as deinterlacing filters, such as nnedi. odd scanlines were interpolated, and since the result was quite nice, it led to use as a general upscaler. anyway, a lot of damage can still happen when trying to go progressive, so my idea is to AI upscale THEN deinterlace to try and eek out as much detail as possible. something like SeperateFields->AiUpscale->Weave->QTGMC. the question is- will the trained model perform when fed half-height photos, or is even more processing required.

[Alexkral]

something like SeperateFields->AiUpscale->Weave->QTGMC. the question is- will the trained model perform when fed half-height photos, or is even more processing required.

From my very limited experience with interlaced content, I'd say that should work fine.

[markiemarcus]

All seems to be working really well. Regarding the dither problem you'd mentioned in the Avisynth Shader thread, does the usual high bit depth approach work around this? For example:

ConvertBits(16)
AiUpscale(luma="vdsr",Chroma="KrigBilateral",mode="illustration")
ConvertBits(8, dither=-1)

[Alexkral]

Processing is done at 16 bit depth anyway, but if the input is 8 bit, the output is converted to 8 bit (without dithering) in the script before returning it. If the input is 16 bit, so is the output, but the transfer to the GPU is slower.

The problem is that AviSynth Shader uses ordered dithering when converting to 8 bit on the GPU before returning the result, so the conversion cannot be done this way which would be faster.

[markiemarcus]

Yes that's definitely a nuisance; there are often times where you need additional processing post upscale and having dither baked like that isn't ideal.

Just to clarify though, passing 16-bit to AIUpscale like this, even though it offers no quality benefits and is slower, doesn't add dither?

[Alexkral]

Dithering is for bit depth reduction only, so no. I could add an option to return 16 bit for an 8 bit input in the next release.

[markiemarcus]

Gotcha, thanks for the clarification. It's actually no real biggie for me given there's some HBD processing going on before and afterwards, but I'm sure that such an extra option might be useful for some.

Many thanks again for your hard work. The results are really impressive and it's considerably faster than every other NN upscaler I've tried. It's just so useful being able to modify the denoising on-the-fly to get the best out of it.