VSGAN - VapourSynth GAN Implementation, based on ESRGAN's implementation

[Selur]

It's relatively fast,...
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/DGDecNV/DGDecodeNV.dll")
# source: 'C:\Users\Selur\Desktop\TheMonkeyKing1965.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 640x480, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\TheMonkeyKing1965.mp4 using DGSource
clip = core.dgdecodenv.DGSource("G:/Temp/mp4_18de9559c23fe1dbb3c03335a2df8593_853323747.dgi")# 25 fps, scanorder: progressive
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0)
from vsanimesr import animesr as AnimeSR
# adjusting color space from YUV420P8 to RGBH for VsAnimeSR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# resizing using AnimeSR
clip = AnimeSR(clip=clip, model=1, device_index=0, trt=True, trt_cache_path=r"G:\Temp") # 2560x1920
# resizing 2560x1920 to 1920x1440
# adjusting resizing
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="limited")
clip = core.fmtc.resample(clip=clip, w=1920, h=1440, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGBS to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
takes:
Output 2510 frames in 60.94 seconds (41.19 fps)
on a Geforce 4080 (not counting the time it takes to build the temporary tensor model) using vspipe.

Cu Selur

[kedautinh12]

Yes with 4080, very fast :D

[PatchWorKs]

For those who are interested in, we recently found this interesting open software:
enhancr is an elegant and easy to use GUI for Video Frame Interpolation and Video Upscaling which takes advantage of artificial intelligence - built using node.js and Electron. It was created to enhance the user experience for anyone interested in enhancing video footage using artificial intelligence. The GUI was designed to provide a stunning experience powered by state-of-the-art technologies without feeling clunky and outdated like other alternatives.

https://camo.githubusercontent.com/b96d020a8183077b8b7b8c9d0b0cdaf4f103060a7bc379a7a98a185d3916dcd2/68747470733a2f2f6d6166696f736e696b2e6e656564736d656e74616c2e68656c702f31774c474837437845412e706e673f6b65793d756b374d3754616f6c4143697071

It features blazing-fast TensorRT inference by NVIDIA, which can speed up AI processes significantly. Pre-packaged, without the need to install Docker or WSL (Windows Subsystem for Linux) - and NCNN inference by Tencent which is lightweight and runs on NVIDIA, AMD and even Apple Silicon - in contrast to the mammoth of an inference PyTorch is, which only runs on NVIDA GPUs.

Features

Encodes video on the fly and reads frames from source video, without the need of extracting frames or loading into memory
Queue for batch processing
Live Preview of output media
Allows chaining of interpolation, upscaling & restoration
Offers the possibility to trim videos before processing
Can load custom ESRGAN models in onnx format
Has Scene Detection built-in, to skip interpolation on scene change frames & mitigate artifacts
Color Themes for user customization
Discord Rich Presence, to show all your friends progress, current speed & what you're currently enhancing
... and much more


Interpolation

RIFE (NCNN) - megvii-research (https://github.com/megvii-research)/ECCV2022-RIFE (https://github.com/megvii-research/ECCV2022-RIFE) - powered by styler00dollar (https://github.com/styler00dollar)/VapourSynth-RIFE-ncnn-Vulkan (https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan)
RIFE (TensorRT) - megvii-research (https://github.com/megvii-research)/ECCV2022-RIFE (https://github.com/megvii-research/ECCV2022-RIFE) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt) & styler00dollar (https://github.com/styler00dollar)/VSGAN-tensorrt-docker (https://github.com/styler00dollar/VSGAN-tensorrt-docker)
CAIN (NCNN) - myungsub (https://github.com/myungsub)/CAIN (https://github.com/myungsub/CAIN) - powered by mafiosnik (https://github.com/mafiosnik777)/vsynth-cain-ncnn-vulkan (unreleased)
CAIN (TensorRT) - myungsub (https://github.com/myungsub)/CAIN (https://github.com/myungsub/CAIN) - powered by HubertSotnowski (https://github.com/HubertSotnowski)/cain-TensorRT (https://github.com/HubertSotnowski/cain-TensorRT)

Thanks to HubertSotnowski (https://github.com/HubertSotnowski) and styler00dollar (https://github.com/styler00dollar) for helping out with implementing CAIN.

Upscaling

waifu2x (NCNN) - nagadomi (https://github.com/nagadomi)/waifu2x (https://github.com/nagadomi/waifu2x) - powered by nlzy (https://github.com/nlzy)/vapoursynth-waifu2x-ncnn-vulkan (https://github.com/nlzy/vapoursynth-waifu2x-ncnn-vulkan)
RealESRGAN (NCNN) - xinntao (https://github.com/xinntao)/Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)
RealESRGAN (TensorRT) - xinntao (https://github.com/xinntao)/Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)

Thanks to HubertSotnowski (https://github.com/HubertSotnowski) for helping out with implementing AnimeSR

Restoration

DPIR (TensorRT) - cszn (https://github.com/cszn)/DPIR (https://github.com/cszn/DPIR) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)


Enjoy: https://github.com/mafiosnik777/enhancr

[kedautinh12]

But, i can't make this app work. Binary here
https://github.com/AmusementClub/enhancr

[ReinerSchweinlin]

Output 2510 frames in 60.94 seconds (41.19 fps)
on a Geforce 4080 (not counting the time it takes to build the temporary tensor model) using vspipe.

Cu Selur
Nice, thanx :)

[lansing]

I set up the Real-ESRGAN trainer to test it out training a model. I only have 11 image pairs and the estimate time is 2 days 6 hours wow, this is discouraging.

[Selur]

out of curiosity: is this mainly eating gpu or cpu resources?

[ChaosKing]

I set up the Real-ESRGAN trainer to test it out training a model. I only have 11 image pairs and the estimate time is 2 days 6 hours wow, this is discouraging.

On some git repo I read that someone trained a model for half a year xD

[lansing]

out of curiosity: is this mainly eating gpu or cpu resources?
GPU I think. I have a Ryzen 7950x and it barely ran.

[ChaosKing]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

[kedautinh12]

Too clean and after use it, i will add some noise to match my favourite :D

[ChaosKing]

You could merge some % of the original clip back to add back some texture.
I just like the overall consistency of the denoising result.

examples of how clean it gets:
https://imgur.com/a/UOjtk8Y
https://imgur.com/a/HB02pCW
https://imgur.com/a/6mQuyIn

[lansing]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

Tested on my anime, the denoising is pretty good. There are some aggressive sharpening when it determined something is an blurred edges. The noise sensitivity is also too high, in my anime here it mistaken the pattern on the wall to be noise.

https://imgsli.com/MTQ0MTkz

[zorr]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

This looks like an excellent prefilter to MVTools, especially for very noisy videos.

I tried to install this to my portable VapourSynth but it looks like vsscunet wants at least Python 3.10 and the latest FatPack is still at 3.9. Are you planning a new release?

[poisondeathray]

scunet is good on gaussian type noise (the pretrained models were trained on synthetic gaussian noise)

non anime example
https://imgsli.com/MTQ0MTk3/0/0

It would be nice if the psnr model had adjustable strength built in.

The gan model retains a bit more detail compared to psnr model; the gan model exhibits less denoising but shifts the colors a bit more

[ChaosKing]

This looks like an excellent prefilter to MVTools, especially for very noisy videos.

I was thinking the same :D

I tried to install this to my portable VapourSynth but it looks like vsscunet wants at least Python 3.10 and the latest FatPack is still at 3.9. Are you planning a new release?

I was actually planning to release a GPU Edition but things got in the way and some laziness. I hope this month :)



The gan model retains a bit more detail compared to psnr model; the gan model exhibits less denoising but shifts the colors a bit more
yeah, color shifts are the most annoying thing with these trained filters.

[lollo2]

I used one of my S-VHS capture (affected by ghosting and halo) with typical noise and the result is very good: deep denoise, but not too much loss of details.
However, I still prefere a classic AviSynth TemporalDegrain2 + LSFmod flow.

Comparison between original and AviSynth: https://imgsli.com/MTI5NjI2

Comparison between original and Scunet: https://imgsli.com/MTQ0MTgw

Comparison between AviSynth and Scunet: https://imgsli.com/MTQ0MTgx

I will try to play with the Scunet options to match AviSynth more "natural" look.

The processing time is very high, >1 minute for a single frame, I hope it will not be like that on a local machine :eek:

[Selur]

Nice.
I agree, looks nice, but it would be nice to have some more control over the denoising strength.
(dot crawlers, light blocking get removed nicely, but general denoising and smoothing seems too strong, probably best applied with some masking)

[zorr]

I agree, looks nice, but it would be nice to have some more control over the denoising strength.

The strength can be somewhat controlled with a couple of tricks. The first one is to amplify the dark areas with gamma (in RGB space) before the denoising and then undoing the gamma change afterwards. It makes the noise look more like actual features so the denoiser leaves more of it intact. The effect is quite subtle though. And it seems to amplify the color drift scunet is doing.


gamma = 1.5
clip = core.std.Levels(clip, gamma=gamma, planes=[0,1,2])
clip = scunet(clip, model = 4)
clip = core.std.Levels(clip, gamma=1.0/gamma, planes=[0,1,2])


Another and more effective way is to upscale the clip before the denoising and rescaling it back to original size afterwards. Here for example with scaling factor 2:


orig_width = clip.width
orig_height = clip.height
clip = core.resize.Lanczos(clip, width=clip.width*2, height=clip.height*2, src_left = -0.25, src_top = -0.25)
clip = scunet(clip, model = 4)
clip = core.resize.Point(clip, width=orig_width, height=orig_height)


Obviously this takes 4 times as much memory and processing power. Scaling by 2 leaves quite a lot more noise so perhaps a smaller scale factor will be enough. And you can of course combine this with the gamma trick.

[lansing]

I'm trying to create a lower quality image from a higher quality image for training a model for anime. I need to recreate a dvd quality image from a cel scan,

from this:
https://imgur.com/FWW2p6B

to this:
https://imgur.com/VdrndTA

Do we have vapoursynth filters that can do that?

[mastrboy]

https://gitlab.com/bg123/vapoursynth-vhs-fx/-/tree/master (I have not tested it)

[lansing]

https://gitlab.com/bg123/vapoursynth-vhs-fx/-/tree/master (I have not tested it)

The syntax was outdated? I'm getting "BlankClip: invalid length" error on this line:

attribute_clip = core.std.BlankClip(clip, length=math.floor(len(clip) * target_fps_num / target_fps_den * clip.fps_den / clip.fps_num), fpsnum=target_fps_num, fpsden=target_fps_den)

[ChaosKing]

Almost perfect decrawl / Derainbow filter 1x_Dotzilla_Compact. https://upscale.wiki/wiki/Model_Database#Anime_and_Cartoons
It just works + speed is also ok.

https://imgsli.com/MTM4ODkz
https://i.imgur.com/hp6tBMC.jpeg

[Selur]

@ChaosKing: nice.

[ReinerSchweinlin]

Almost perfect decrawl / Derainbow filter 1x_Dotzilla_Compact. https://upscale.wiki/wiki/Model_Database#Anime_and_Cartoons
It just works + speed is also ok.

https://imgsli.com/MTM4ODkz
https://i.imgur.com/hp6tBMC.jpeg
looking good :)

[PatchWorKs]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

Nice, here's a "degrain" of a 1080p source (model - real image denoise):
https://i31.servimg.com/u/f31/16/32/04/85/vlcsna11.jpg
https://i31.servimg.com/u/f31/16/32/04/85/output10.jpg

Definition and shadows looks ok, blurry is minimal.

Anyway I can notice a little banding (check the sky in the center of the image).

EDIT: https://imgsli.com/MTUxMTY5 (overlay comparison)

[ChaosKing]

yeah its very good. But look at the guys shirt. It's like a "shadow" is removed or the shirt got wasched clean. Interessting effect :D

I wonder if a "temporal scunet" would be even better.

If think if you combine the scunet filtered frame with a smdegrain filtered frame the result would be almost perfect (maybe add some debanding too)

[PatchWorKs]

...it would be also interesting to have a "combined" (denoise+SDRtoHDR) tool:

https://github.com/MCG-NKU/FMNet

It could be also *very* useful to perform a block partitioning-optimization at the same time:

https://github.com/fengxinmin/iDepthmap-x265

[PatchWorKs]

Just discovered aydin (https://royerlab.github.io/aydin/), "a user-friendly, feature-rich, and fast image denoising tool that provides a number of self-supervised, auto-tuned, and unsupervised image denoising algorithms":

https://royerlab.github.io/aydin/v0.1.15/_images/aydin_welcome_screen.png
https://royerlab.github.io/aydin/v0.1.15/_images/aydin_denoise_tab.png

Website: https://royerlab.github.io/aydin/

Git: https://github.com/royerlab/aydin

Quite surprised that we didn't found any related search result here ad Doom9...

[Selur]

Not that surprising, since there are not many threads that are image centric.
Usually it's about video. And image based filters can easily lead to temporal inconsistencies.

[PatchWorKs]

Not that surprising, since there are not many threads that are image centric.
Usually it's about video. And image based filters can easily lead to temporal inconsistencies.
Very interesting reply by aydin's author (https://github.com/AhmetCanSolak) @ my GitHub "issue" (https://github.com/royerlab/aydin/issues/292):
I want to clarify that Aydin works on spatio-temporal data(TZYX) when all four dimensions are available and can do TYX as well. Aydin doesn't read video file formats directly though, if you can save it as a file format that is supported then you can denoise movies too. With biological live imaging datasets, we use Aydin for denoising movies on daily basis.

[PatchWorKs]

@Selur and anyone else interested in participating are called to collaborate by "make bug reports/feature requests here on GitHub" according to the aydin's author:
https://github.com/royerlab/aydin/issues/292#issuecomment-1428538108

[Selur]

btw. has anyone tested if there are speed differences depending on the RGB color space VSGAN is fed?
Just noticed that for me (only testst a few models), RGBS seems to be ~10% faster than when I feed any other RGB color space.
-> Is it just me or can others confirm this?

Cu Selur

[poisondeathray]

btw. has anyone tested if there are speed differences depending on the RGB color space VSGAN is fed?
Just noticed that for me (only testst a few models), RGBS seems to be ~10% faster than when I feed any other RGB color space.
-> Is it just me or can others confirm this?

Cu Selur

Similar for me ; RGBS was about 7-10% faster than RGB24 (source was RGB24)

[Selur]

Thanks for testing :)

[Selur]

Argh, I got a strange problem when using vsdpir and vsgan inside the same script:

using both with RGH:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

works.

using both with RGBS:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

works.

Using VSDIR with RGBS and VSGAN with RGBH:
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBS to RGBH for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, range_s="full", dither_type="error_diffusion")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip
works.

Using VSDIR with RGBH and VSGAN with RGBS:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBH to RGBS for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="full")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

fails with:
Input type (float) and bias type (struct c10::Half) should be the same


Using VSDIR with RGBS and VSGAN with RGB48:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBS to RGB48 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, range_s="full", dither_type="error_diffusion")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

produces graphic glitches here https://ibb.co/jTCbghX

Cu Selur

Ps.: I also reported this to https://github.com/rlaphoenix/VSGAN/issues/31, but thought it might be interesting to others in case they run into the same issue.

[Selur]

Okay, issue seems to be that VSDPIR overwrites the default tensor type. (so not a bug in VSGAN, but an unexpected behavior of VSDPIR)
Using:
import torch
torch.set_default_tensor_type(torch.FloatTensor)
after VSDPIR, should fix this it.

Behavior is fixed with https://github.com/HolyWu/vs-dpir/releases/tag/v3.0.1

Cu Selur

[lansing]

New 2x upscaling model for anime AniScale2
https://github.com/Sirosky/Upscale-Hub/releases/tag/AniScale2

It only works for digital created animes. It came with a base upscale model and a refiner model for sharpening. The model has depth of field awareness so the intended blur effect will be kept intact in the result.

I tested it a little on BLEACH and it looks pretty convincing.

[Selur]

Interesting, thanks for the info. :)