ONNX models to use with avs-mlrt [Archive]

Reel.Deel

26th February 2023, 03:43

PyTorch models converted to ONNX to use with AviSynth+'s avs-mlrt plugin (https://github.com/Asd-g/avs-mlrt). Models were converted with chaiNNer (https://github.com/chaiNNer-org/chaiNNer) (fp32 data type). Let me know of missing models to convert and I'll add them to the package.

Download: onnx-models-rd (https://www.dropbox.com/sh/f74ao9t1qarffok/AACfLoG7-F8MvWSfwifgr6M4a?dl=0)

Other repositories that include onnx models:

Large collection of onnx models: https://github.com/styler00dollar/VSGAN-tensorrt-docker/releases/tag/models

2x_AnimeJaNai - https://github.com/the-database/mpv-upscale-2x_animejanai/

Phhofm models (some already onnx): https://github.com/Phhofm/models

Model database

https://upscale.wiki/wiki/Model_Database

https://openmodeldb.info/

Basic usage
FFVideoSource("YUV420Source.mkv")
ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="1x_Dotzilla_Compact_80k_net_g.onnx", builtin=false)
You can omit the builtin parameter if the models are in the same location as the avs-mlrt plugin (see builtin and builtindir parameters in the docs).

If you have a low end GPU and trying to use the larger models, it helps to use the tilesize, overlap and fp16 options.
mlrt_ncnn(network_path=model, builtin=false, fp16=true, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)

Model conversion

Download chaiNNer (https://github.com/chaiNNer-org/chaiNNer) and install the required dependencies. Load the corresponding nodes, connect them and then hit run by either pressing the play button at the top or F5.

https://i.ibb.co/6P4DDMg/chainner-convert.png

Dogway

26th February 2023, 03:57

kedautinh12

26th February 2023, 04:02

wow, so i think this will open new era about pytorch can port to avisynth

kedautinh12

26th February 2023, 04:03

Reel.Deel

26th February 2023, 05:06

Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Old Anime Upscale:
1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
4x_eula_digimanga_bw_v1_860k.pth
Magazines?
4x-UniScaleV2_Moderate

Thanks for the suggestions. I added them to the package. I'll provide a Dropbox link from now on since the size is already at 331mb. This way people can pick and choose only the models they want. I did not test these new models, let's hope they don't blow up :)

Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).

poisondeathray

26th February 2023, 05:17

Can it use for real world video??

Usually the models are quite narrow in scope; typically they are only useful on the data set that they were trained on, or very similar material. But you can try it...

Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).

Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"

Reel.Deel

26th February 2023, 05:19

Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"

I guess the latest one. There is no version number on the upscale wiki.

poisondeathray

26th February 2023, 05:22

Reel.Deel

26th February 2023, 05:33

I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoqomhrt88bsk/NMKD+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?

Thanks for the models.

1) Depends on the size of the model. Small ones are quick (few seconds), the largest one I converted (64mb) took around a minute.

2) I have not compared them to the original version. It would be interesting to do so. But while researching how to convert between the two formats I only saw that mentioned once or twice. See here: https://www.reddit.com/r/MachineLearning/comments/hqqymm/dhow_much_can_converting_a_torch_model_to_onnx/ and here https://pytorch.org/docs/stable/onnx.html#limitations

I'm all ears if anyone has any suggestions.

This is the interface in ChaiNNer:
https://i.ibb.co/RNTcP0p/chainner.png
I'm sure there more options when using PyTorch directly but I have not experimented with that.

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Arx1meD

26th February 2023, 09:00

This plugin works on GPU only?

DTL

26th February 2023, 11:56

For denoise process (in physical temporal dimension) we need models for motion compensation. The framerate interpolation models are only partially good for this processing. Better neural network engines for motion compensation need accept several input frames in a sequence (not only 2 frames as we have in RIFE now).

The 'training condition' for motion compensation process is simple enough: For given number of frames with marked 'current' frame it is required to create 'current' frame view using only 'other' frames. In simplest form with 2 frame only input - we have only 'current' and 'other' frame. So engine need to reconstruct 'current' frame using only samples data from 'other' frame (it can use 'current' frame only as 'objects positions reference').

It may be even more simple in compare with 'framerate conversion' process where engine have 2 frames 'before and after' and 'current' frame to reconstruct at time of model training.

Can it use for real world video??

One issue I found when trying to use online AI upscalers - they may not understand the depth of field blur and start to attempt to make out of focus objects sharper too. So for simple 2D drawing without textures like anime it work for sharp lines restoration from aliasing and all frame deblur about good. But for 3D real world lens projection to 2D buffer + different type (and DOF blur) textures it may fail significantly.

Dogway

26th February 2023, 15:34

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Didn't crash for me, but the GPU got high rolling for a few seconds... scared.

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.

anton_foy

26th February 2023, 16:10

DTL: The 'training condition' for motion compensation process is simple enough: For given number of frames with marked 'current' frame it is required to create 'current' frame view using only 'other' frames. In simplest form with 2 frame only input - we have only 'current' and 'other' frame. So engine need to reconstruct 'current' frame using only samples data from 'other' frame (it can use 'current' frame only as 'objects positions reference').

It may be even more simple in compare with 'framerate conversion' process where engine have 2 frames 'before and after' and 'current' frame to reconstruct at time of model training.

Tried to find any model for this kind of motion compensation but without success. One would think there would be some out there? Although I do not specifically know what words to use for searching. I tried "model pytorch motion compensation" but mostly got hits about object tracking and similar.

Edit: if someone knows how to do the training bit I can provide with before/after tests in UHD (Sony does 6k then downscales internally to 3840x2160 and it is so crisp with minimal halo when internal sharpness is turned off).

poisondeathray

26th February 2023, 17:14

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

There are some inpainting algos that use different architectures (cannot be run in avs/vpy/cupscale)

https://github.com/geekyutao/Image-Inpainting
https://github.com/zengyh1900/Awesome-Image-Inpainting

The problem I found with single image/spatial inpainting algos, is the flicker and temporal inconsistencies (you'd have to apply some temporal smoothing after)

Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo

Chainner lists support for 2 spatial algos that I haven't tried yet; but they should "work" since the GUI has direct support

Lama
https://github.com/advimman/lama

MAT
https://github.com/fenglinglwb/MAT

Tried to find any model for this kind of motion compensation but without success. One would think there would be some out there? Although I do not specifically know what words to use for searching. I tried "model pytorch motion compensation" but mostly got hits about object tracking and similar.

Object tracking is part of it. Also propogation and alignment.

There are typically catergorized into single image vs. video . "Video" algorithms takes >1 frame into account. All the ESRGAN / Real-ESRGAN derivatives are single image only.

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

I posted some examples of BasicVSR/BasicVSR++ in a few threads here and other forums. The provided model works on "textbook" clean cases. A big problem for is degradation step wasn't included , so it's not as generalizable to real world, compression cases. RealBasicVSR adds degradation, but the results are quite blurry / detail loss

poisondeathray

26th February 2023, 17:22

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Could be GPU memory ?

Did you try the tiling options ?

- overlap_w, overlap_h\
Overlap width and overlap height of the tiles, respectively.\
Must be less than or equal to `tilesize_w` / `tilesize_h` `/` `2`.\
Default: 0.

- tilesize_w, tilesize_h\
Tile width and height, respectively.\
Use smaller value to reduce GPU memory usage.\
Must be specified when `overlap_w` / `overlap_h` > 0.\
Default: input_width, input_height.

And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...

DonCanjas

26th February 2023, 17:34

Would like to see

4x_eula_digimanga_bw_v2_nc1_307k
MangaJPEG
1x_GainRESV3 (Aggro,Natural,Passive)
1x_DitherDeleter-Smooth_104000_G

kedautinh12

26th February 2023, 17:48

@VX will happy if had delogo models work perfectly:D

poisondeathray

26th February 2023, 17:57

And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...

Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about

poisondeathray

26th February 2023, 18:01

This plugin works on GPU only?

I think so, the requirements say Vulkan device. I do not see a "CPU" mode anywhere

https://github.com/Asd-g/avs-mlrt

DTL

26th February 2023, 18:05

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

And what is the current RIFE is based at ? Or it is separated engine for doing motion interpolation ? Some wiki https://www.svp-team.com/wiki/RIFE_AI_interpolation says it is based on IFNet neural network.
Also it shows TensorRT at NVIDIA cards can run RIFE about 2x faster in compare with ncnn/Vulkan implementation. So RIFE (IFNet ?) can be compiled (?) to different hardware APIs ?

As for only storing frames in memory - the 1920x1080 frame in 12bytes per sample RGB 32bit float is about 24 MB only. So 1/4 of 1GB onboard RAM can keep 10 frames and it is very good already to study how motion is going between several frames.

I post an issue on some RIFE development project at github about possible changing RIFE algorithm for better denoise process - https://github.com/megvii-research/ECCV2022-RIFE/issues/304 but still no one reply. May be it require to search someone with understanding how RIFE work so the required changes can be implemented.

kedautinh12

26th February 2023, 18:07

Can ask Asd-g about RIFE??

Dogway

26th February 2023, 18:09

Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo

Thanks a lot, yes I wonder why temporal information isn't used more frequently in inpainting. I mean it has huge potential not only in restoration but also VFX. I will give it a stab when I'm in Win10.

poisondeathray

26th February 2023, 18:27

And what is the current RIFE is based at ? Or it is separated engine for doing motion interpolation ?

I don't understand the question - RIFE is flow based frame prediction (intermediate frame(s) synthesized between 2 source frames) .

The video SR/deblurring/denoising algos mentioned are multiframe bidirectional ...n-3, n-2, n-1, n, n+1, n+2, n+3... you can set 30, 50 or more frames if you wanted to

Some wiki https://www.svp-team.com/wiki/RIFE_AI_interpolation says it is based on IFNet neural network.
Also it shows TensorRT at NVIDIA cards can run RIFE about 2x faster in compare with ncnn/Vulkan implementation. So RIFE (IFNet ?) can be compiled (?) to different hardware APIs ?

Yes, different implementations are possible

Yes, TensorRT is usually the fastest implementation for all machine learning algorithms, not just rife .

As for only storing frames in memory - the 1920x1080 frame in 12bytes per sample RGB 32bit float is about 24 MB only. So 1/4 of 1GB onboard RAM can keep 10 frames and it is very good already to study how motion is going between several frames.

GPU memory, not system memory . And it's not just the additive frame memory; pytorch /TensorRT/ NCNN, all of them - consume extra reserve memory for operations .

I post an issue on some RIFE development project at github about possible changing RIFE algorithm for better denoise process - https://github.com/megvii-research/ECCV2022-RIFE/issues/304 but still no one reply. May be it require to search someone with understanding how RIFE work so the required changes can be implemented.

I haven't had a chance to look at it yet, but I'm not sure that RIFE would add much over for the mocomp over say mvtools2, and it would be slower

anton_foy

26th February 2023, 18:34

Object tracking is part of it. Also propogation and alignment.

There are typically catergorized into single image vs. video . "Video" algorithms takes >1 frame into account. All the ESRGAN / Real-ESRGAN derivatives are single image only.

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

I posted some examples of BasicVSR/BasicVSR++ in a few threads here and other forums. The provided model works on "textbook" clean cases. A big problem for is degradation step wasn't included , so it's not as generalizable to real world, compression cases. RealBasicVSR adds degradation, but the results are quite blurry / detail loss

Thanks for the thorough reply. I tried DPIR but had to downscale from uhd to 1280x720 for it to run and then it runs pretty fast but runs not at all above that resolution.

Even if it is only "spatial" denoising it can be used before MSuper() spatialfilter.MSuper()
and then Mdegrain to "smooth" out the motion jitter.
Usually I do this with fft3dfilter and mdegrain but maybe there is some better way to stabilize the motion jitter/wobble caused by noise.

Very curious to try the noise models in the model_database link REEL.DEEL posted, especially the "1x_ISO_denoise_v1" and v2 and "Film-Degrainer_1-000".

Reel.Deel

26th February 2023, 18:37

Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about

Thanks for testing, good to know that there are no big differences. That's the consensus I got from other places. When converting some models I get a warning "Backend: WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function." but from what I read (https://github.com/ultralytics/yolov5/issues/6401), they are harmless.

Could be GPU memory ?

Did you try the tiling options ?

Probably, I need to experiment with those options.

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.

I'll add those to the package.

Would like to see

4x_eula_digimanga_bw_v2_nc1_307k
MangaJPEG
1x_GainRESV3 (Aggro,Natural,Passive)
1x_DitherDeleter-Smooth_104000_G

Will do.

DTL

26th February 2023, 18:37

Can ask Asd-g about RIFE??

May be only RIFE developers https://arxiv.org/pdf/2011.06294v12.pdf understand how it is designed and how to change in into motion compensating engine based on reference frame. Not to interpolate between 2 given images.

Asd-g only make interface between RIFE engine and AVS+.

poisondeathray

26th February 2023, 18:41

Thanks for the thorough reply. I tried DPIR but had to downscale from uhd to 1280x720 for it to run and then it runs pretty fast but runs not at all above that resolution.

You can use "tiling" .

Look at the avsi for tilesize and overlap options. For spatial denoising like DPIR, if you use large enough overlap, the differences are usually undetectable to eye (except with amplified differences) between single tile (full image), vs. say 2x2 grid - many tests were done with the vapoursynth version in the past - I'm assuming the avs version is the same

Even if it is only "spatial" denoising it can be used before MSuper() spatialfilter.MSuper()
and then Mdegrain to "smooth" out the motion jitter.
Usually I do this with fft3dfilter and mdegrain but maybe there is some better way to stabilize the motion jitter/wobble caused by noise.

Not just for denoising; all single image operations are prone temporal inconsistencies. Another option is QTGMC in progressive mode but it has downsides too, even if you adjust the settings

It would be nice to have other options for temporal smoothing

DTL

26th February 2023, 18:58

The video SR/deblurring/denoising algos mentioned are multiframe bidirectional ...n-3, n-2, n-1, n, n+1, n+2, n+3... you can set 30, 50 or more frames if you wanted to

I haven't had a chance to look at it yet, but I'm not sure that RIFE would add much over for the mocomp over say mvtools2, and it would be slower

Do these algos mentioned have frame interpolation implementations (like framerate doubling, same as RIFE) so can also be tested for denoise performance/quality ?

The major issue of current motion compensation in mvtools is being limited to compensation of translate transform only. Also its motion estimation is not ideal because of noise (and noise can not be removed before motion estimation - it is 'egg and chicken' problem same as mentioned in that RIFE document).

So from the neural networks denoise (motion compensation for 'static' blending engines) expected:
1. Compensate for more real transforms (scale/rotate/skew/...)
2. To be less sensitive to natural noise in the sources to analyse (using variable size / variable shape areas to analyse in a big enough sequence of frames or better in total scene or even movie).

Current mvtools approach to decrease block size to have better and finer motion tracking also degrades its ability to correctly track noise-damaged blocks. So when signal-to-noise ratio come close to 1 or fall below 1 typical block-based motion search engines (also limited to 2 frames only) give very large errors and mostly useless.

So if in current quickly dying from degradation civilization we have very limited and dying programmers resources to design better motion search engines we try to look for 'hardware/machine learning' tools that can replace current poor programmers resources by 'brute force' of GBytes memory and TeraFlops+ accelerators and 'neural networks with ability to learn'. The performance is not very critical if it can provide significantly better quality in compare with current block-based simple motion search in mvtools. Some titles wait years and decades before being processed in yet another rip for release in possibly more or less better quality. So if it even will be processed several days/weeks/months it is not real physical problem.

poisondeathray

26th February 2023, 19:07

Do these algos mentioned have frame interpolation implementations (like framerate doubling, same as RIFE) so can also be tested for denoise performance/quality ?

VRT does; it's very very slow , and consumes large amounts of memory . It has 3 tiling , 3 overlap parameters, frame(temporal), height, width . (note that it's height first, not width)

https://github.com/JingyunLiang/VRT

The major issue of current motion compensation in mvtools is being limited to compensation of translate transform only. Also its motion estimation is not ideal because of noise (and noise can not be removed before motion estimation - it is 'egg and chicken' problem same as mentioned in that RIFE document).

Prefilter can sometimes help, but yes chicken vs. egg.

DTL

26th February 2023, 19:34

Prefilter can sometimes help, but yes chicken vs. egg.

Prefilter may degrade some valueable details for output. Also prefilter may increase motion search error. So I not like to use 'simple' prefilter before motion-compensated averaging.

"https://github.com/JingyunLiang/VRT"

It looks VRT still not have interface to Avisynth ? And can not be loaded via AviSynth+'s avs-mlrt plugin ?

poisondeathray

26th February 2023, 19:48

It looks VRT still not have interface to Avisynth ? And can not be loaded via AviSynth+'s avs-mlrt plugin ?

No avs interface, and cannot be loaded with avs-mlrt

Reel.Deel

27th February 2023, 00:05

Uploaded a handful of models that were requested here and also a few other's of Zarxrax's models (2x_LD-Anime_Compact_330k_net_g, 1x_Dotzilla_SuperUltraCompact_300k_net_g, 1x_Dotzilla_UltraCompact_182k_net_g).

@dogway, 1x_NMKD-YandereInpaint_375000_G is not available, do you have DL link?

Edit

https://www.mediafire.com/file/fp8gdb2a6ltdzra/NMKD+Yandere+Inpaint.7z/file

Thanks!

poisondeathray

27th February 2023, 00:10

1x_NMKD-YandereInpaint_375000_G is not available, do you have DL link?

https://www.mediafire.com/file/fp8gdb2a6ltdzra/NMKD+Yandere+Inpaint.7z/file

kedautinh12

27th February 2023, 02:05

poisondeathray

27th February 2023, 02:15

can you convert it to onnx??

https://www.mediafire.com/file/51maete86cv48ib/2x_NMKD-DeGIF_210000_G_onnx.7z/file

kedautinh12

27th February 2023, 02:32

Thanks

Reel.Deel

27th February 2023, 02:52

Could be GPU memory ?

Did you try the tiling options ?

I was able to get it running using those options, although at a snails pace. I guess my GPU is not adequate for the bigger models (1050 now, I plan on upgrading to a 2070 in the upcoming months).

kedautinh12

27th February 2023, 05:22

All avs+ plugins collection by me, sort by architecture like Asd-g. @Reel.Deel i think you can sort like that for easy findout
https://drive.google.com/file/d/11Vk0HB58G7qVjKK0ZPRkemUnI1iAG71y/view?usp=drivesdk

anton_foy

27th February 2023, 09:16

You can use "tiling" .

Look at the avsi for tilesize and overlap options. For spatial denoising like DPIR, if you use large enough overlap, the differences are usually undetectable to eye (except with amplified differences) between single tile (full image), vs. say 2x2 grid - many tests were done with the vapoursynth version in the past - I'm assuming the avs version is the same

Not just for denoising; all single image operations are prone temporal inconsistencies. Another option is QTGMC in progressive mode but it has downsides too, even if you adjust the settings

It would be nice to have other options for temporal smoothing

Thanks! Yes tiling and overlap, I need to get back to my comp and try this, I had good results too with QTGMC's denoising in the past but it left some noise yet calmed down. Could you or someone convert 1x_ISO_denoise_v2? I would like to try all of the denoisers/degrainers in there but if that is too much I can try to convert them myself. Any link to the conversion software or does it cost?

Edit: thank you so much Reel.Deel great to see these kind of filters coming out for avs+!

Dogway

27th February 2023, 12:26

kedautinh12

27th February 2023, 13:37

About gain, maybe this model can help you "1x_Filmify4K_v2_325000_G"

Reel.Deel

27th February 2023, 15:39

Thanks for the conversions Reel.Deel!
Maybe HiNet can be converted? It had really astonishing (https://forum.doom9.org/showthread.php?p=1964013#post1964013) results for scratches.
And just asking out of the blue, not sure if a film grain model as such exists, that is to add filmic grain, it can be an alternative to chickendream which is too slow. EDIT: yes, found 1x_Filmify4K_v2_325000_G but grain is not of great quality...
I also had 4x_Valar_v1 which is good for photo upscales it seems, but as with many of these it might require perfect artifact free downscaled content.

Also as a suggestion it might be interesting to try 2x_Waifaux-NL3-SuperLite_latest_G, as it might be a faster alternative to current solutions like w2xncnnvk.

No problem, I will add your other suggestions (edit: added). And HiNet cannot be converted (at least in ChaiNNer, who knows if there are other methods).

EDIT: Tested 1x_JPEG_60-80 and 1x-Kim2091_DeJpeg and got a system freeze, black screen and AvsPmod crash. Well anyway I compared them to Oyster and they paled against it. Since Oyster is so slow, it would be interesting to train a model based on it with my script from this post (https://forum.doom9.org/showthread.php?p=1981849#post1981849) and probably a deblur model on it. Maybe if I have time in the future I can do such thing.

What were the dimensions of the image when trying 1x_JPEG_60-80 and 1x-Kim2091_DeJpeg? For me, I can't get most of the bigger models to run (black screen, driver crash) unless I use the tilesize and overlap options. With Kim2091_DeJpeg I had no trouble with but only used it on small picture. And I agree, I was not too impressed.

Training a model on Oyster does sound interesting. I read that Zarxrax used avs filters to train some of his models.

Could you or someone convert 1x_ISO_denoise_v2?
Done.

Dogway

27th February 2023, 16:14

The same image from the linked post, 472x400 pixels. I will try the tiling option. I mean my GPU is not THAT bad : ) TBH I'm waiting for an affordable +10Gb card, be it this gen or the next one, the xx70 sitting on 8Gb for 8 years is as ridiculous as the current prices.

poisondeathray

27th February 2023, 17:04

I had good results too with QTGMC's denoising in the past but it left some noise yet calmed down.

I wasn't referring to QTGMC for the purpose of denoising; I was referring to using single image filters on video such as 99% of the ones in that database - they can be prone to temporal inconsistencies. QTGMC inputtype=1 or 2 is one option that can be used after the single image filter(s) that can smooth over the consistencies, but as you know there are side effects to using QTGMC in progressive mode

Any link to the conversion software or does it cost?

chaiNNer is free. Reel.Deel posted instructions how to convert earlier.
https://github.com/chaiNNer-org/chaiNNer/releases

Reel.Deel

27th February 2023, 17:17

It's very easy to convert also. Load the required nodes, connect them and then hit run by either pressing the play button at the top or F5. Just make sure you've installed the dependencies.

https://i.ibb.co/6P4DDMg/chainner-convert.png

Side note for Win7 users like me. Alpha v0.17.1 is the last version that works and you have to place the api-ms-win-core-path-l1-1-0.dll (https://github.com/nalexandru/api-ms-win-core-path-HACK) in the Python folder which appears after the first time starting the program.

anton_foy

27th February 2023, 21:33

Thanks Reel.Deel!

Quote:
Originally Posted by anton_foy View Post
I had good results too with QTGMC's denoising in the past but it left some noise yet calmed down.
I wasn't referring to QTGMC for the purpose of denoising; I was referring to using single image filters on video such as 99% of the ones in that database - they can be prone to temporal inconsistencies. QTGMC inputtype=1 or 2 is one option that can be used after the single image filter(s) that can smooth over the consistencies, but as you know there are side effects to using QTGMC in progressive mode

Quote:
Any link to the conversion software or does it cost?

chaiNNer is free. Reel.Deel posted instructions how to convert earlier.
https://github.com/chaiNNer-org/chaiNNer/releases

Aah okay I had too little knowledge of QTGMC I did not know about this mode. Thanks for the link, I will use it!

DTL

9th March 2023, 10:58

Prefilter can sometimes help, but yes chicken vs. egg.

Finally iterative solution for chicken and egg problem in motion compensating denoising - https://forum.doom9.org/showthread.php?p=1984152#post1984152

Now I think if it possible to replace either MAnalyse or MDegrain in that solution with some neural network processing from existing but keeping the iterative idea of checking each generation result with non-distorted input source.

In each iteration the algorithm refines both denoised clip and MVs field. But there is no current NN plugins capable of return MVs field in compatible with mvtools format ?

May be instead of MDegrain create an interleaved clip with MCompensated frames from input MVclip for each iteration and pass it to RIFE denoise ?

Reel.Deel

18th March 2023, 20:53

I'll respond here to not hijack Selur's thead (https://forum.doom9.org/showthread.php?t=184797).

No error message ? Just "hangs" ?

Yet other onnx conversions work ok on that same avs setup ?

Did you check that other applications which might be using GPU are closed ?

I closed everything, except VDub2 and it just hangs. I let it run for about 5 minuted and nothing. It says not responding and shows this:

Description:
A problem caused this program to stop interacting with Windows.

Problem signature:
Problem Event Name: AppHangB1
Application Name: VirtualDub64.exe
Application Version: 2.0.0.0
Application Timestamp: 5e73f48a
Hang Signature: 315e
Hang Type: 0
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1033
Additional Hang Signature 1: 315e1db675843581067ed78f1c415427
Additional Hang Signature 2: 4ae5
Additional Hang Signature 3: 4ae5c6bd4a564e63c8d65dcb154a1aa2
Additional Hang Signature 4: 315e
Additional Hang Signature 5: 315e1db675843581067ed78f1c415427
Additional Hang Signature 6: 4ae5
Additional Hang Signature 7: 4ae5c6bd4a564e63c8d65dcb154a1aa2

I have not tried many models but all of Zarxrax's models work fine and also 1x_SwatKatsLite_360000_G.

Edit: It works now due to something very odd, this is my script:

FFVideoSource("HT1-1-RainbowsSample.m2v") # NTSC DVD
TFM(PP=0)
TDecimate()
ConvertToPlanarRGB()
o = last
ConvertBits(32)
mlrt_ncnn(network_path="...\2x_AnimeClassics_UltraLite_510K.onnx", builtin=false, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)
ConvertBits(8)
#Interleave(o,last)

When Interleave(o,last) is not commented out, it just hangs :confused:

StvG

19th March 2023, 06:09

FFVideoSource("HT1-1-RainbowsSample.m2v") # NTSC DVD
TFM(PP=0)
TDecimate()
ConvertToPlanarRGB()
o = last
ConvertBits(32)
mlrt_ncnn(network_path="...\2x_AnimeClassics_UltraLite_510K.onnx", builtin=false, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)
ConvertBits(8)
#Interleave(o,last)

When Interleave(o,last) is not commented out, it just hangs :confused:

"o" and "last" have different dimensions.

Selur

19th March 2023, 08:25

"o" and "last" have different dimensions.
I agree, but it should complain about that and not crash,...

Reel.Deel

19th March 2023, 16:53

"o" and "last" have different dimensions.

My bad, I completely overlooked that. :stupid:

But there might be something else going on. Interleave always throws an error when the dimensions are different. Can anyone else reproduce this behavior?

poisondeathray

19th March 2023, 17:25

But there might be something else going on. Interleave always throws an error when the dimensions are different. Can anyone else reproduce this behavior?

Just with a "normal" script ?

a=colorbars()
b=a.bicubicresize(a.width*2, a.height*2)

interleave(a,b)

Interleave: videos must be of the same size.

Reel.Deel

19th March 2023, 17:38

Just with a "normal" script ?

With a "normal" script it works fine. But when avs-mlrt is involved and the interleaved videos are different dimensions, it causes VDub2 to hang. At least for me. Can you try this script:

ColorBars(pixel_type="RGBPS")
o = last
mlrt_ncnn("2x_AnimeClassics_UltraLite_510K.onnx", builtin=false)
Interleave(o,last)
ConvertBits(8)

It works fine if o = last.PointResize(width*2, height*2).

poisondeathray

19th March 2023, 17:45

Can you try this script:

ColorBars(pixel_type="RGBPS")
o = last
mlrt_ncnn("2x_AnimeClassics_UltraLite_510K.onnx", builtin=false)
Interleave(o,last)
ConvertBits(8)

I get the "Interleave: videos must be of the same size. " message in avspmod; but vdub2 hangs

Reel.Deel

19th March 2023, 17:50

I get the "Interleave: videos must be of the same size. " message in avspmod; but vdub2 hangs

Same. I guess it's a VDub2 issue.

takla

20th March 2023, 02:29

Fails silently with my AMD Radeon GPU...
Standalone vulkan ncnn converters do work. So the culprit must be the avisynth plugin.

poisondeathray

20th March 2023, 02:38

Fails silently with my AMD Radeon GPU...
Standalone vulkan ncnn converters do work. So the culprit must be the avisynth plugin.

Did you fulfill the requirements in the ReadMe ?

Other common "culprits" for some people's silent fails were non meeting the avs+ minimum version, and GPU memory (use tiles) . Neither current produces a useful error message, which could be an improvement

takla

20th March 2023, 09:47

Did you fulfill the requirements in the ReadMe ?

Other common "culprits" for some people's silent fails were non meeting the avs+ minimum version, and GPU memory (use tiles) . Neither current produces a useful error message, which could be an improvement

Lmao. I literally updated everything but avisynth+. That was the issue. Works now with r3958. Thank you ;)

Dogway

26th March 2023, 00:37

Reel.Deel

27th March 2023, 00:14

I found new interesting models, not sure how feasible is it to convert to onnx, compatibility-wise:
Photos:
Real ESRGAN 4x Plus
Real ESRGAN General WDN 4x V3 (photos, paintings or anime illustrations)
Latent Diffusion Super Resolution (LDSR) (SUPER slow though)
SwinIR
Swin2SR
Remacri
Anime/LineArt Illustration Upscale:
Real ESRGAN 4x Plus Anime6B

Do you have a link to Real ESRGAN 4x Plus and LDSR pth models?

I did not test them, but so far these models convert to to onnx without any problems:

realesr-general-wdn-x4v3.pth : https://github.com/xinntao/Real-ESRGAN/releases/tag/v0.3.0

RealESRGAN_x4plus_anime_6B.pth : https://github.com/xinntao/Real-ESRGAN/blob/master/docs/anime_model.md

Remacri : https://upscale.wiki/wiki/Model_Database#Universal_Models

SwinIR : https://github.com/JingyunLiang/SwinIR/releases

What specific SwinIR model you want converted, or all?

With the Swin2SR models (https://github.com/mv-lab/swin2sr/releases) I get an error when trying to covert:
An error occurred in a Convert To ONNX node:

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument tensors in method wrapper___cat)

Input values (partial):
• Data Type: fp32

Not sure what that means.

Dogway

27th March 2023, 00:28

Most models are posted here (https://github.com/xinntao/Real-ESRGAN/blob/master/docs/model_zoo.md).
Interested on realesrgan-x4minus as well, but the link is broken on the model_data page, I found a link in a reddit post, searching again now.

These are the Swin2SR models, the recommended for compressed jpeg photos. Haven't tested them, but as I could see Jpeg_dynamic seems the best, probably the others are also worth having a look:
Swin2SR_Jpeg_dynamic.pth
Swin2SR_ClassicalSR_X2_64.pth
Swin2SR_ClassicalSR_X4_64.pth
Swin2SR_CompressedSR_X4_48.pth
Swin2SR_Lightweight_X2_64.pth

Here (https://phhofm.github.io/upscale/favorites.html) they recommend SwinIR-L with CodeFormer (https://github.com/sczhou/CodeFormer#quick-inference) (faces) or GFPGAN (https://github.com/TencentARC/GFPGAN#european_castle-model-zoo) and v1.4 (https://github.com/chaiNNer-org/chaiNNer#face-restoration) (also faces), might be worth to check out.

About LDSR I found a post from here (https://www.reddit.com/r/StableDiffusion/comments/yev37i/comparison_of_upscaling_models_for_ai_generated/)that says:
chaiNNer does not support LDSR, but you can use it for example on replicate:
https://replicate.com/nightmareai/latent-sr

EDIT: You can find 4xLDSR now here (https://github.com/Phhofm/models/tree/main/4xLSDIRCompact/Version3) as onnx. The C version is good for heavy blocking images.

By the way, I wasn't aware either, but the wiki has also a page for "official" models which includes many of the ones listed above: https://upscale.wiki/wiki/Official_Research_Models

DTL

29th March 2023, 21:29

poisondeathray

29th March 2023, 21:51

There is some promising update from neural-networks designers (authors of RIFE) about 'frame prediction' - https://github.com/megvii-research/CVPR2023-DMVFN . And recommended to check. It is expected to be better motion compensation engine in compare with current RIFE used in temporal denoising.

It uses 2 past frames to predict next frame

I posted some examples in this thread.

https://forum.doom9.org/showthread.php?t=184387

Can it be used in AVS via existing plugin or require plugin redesign ?
Not currently.

If someone makes a ncnn/vulkan compatible version then possibly avs version could materialize. None of the direct pytorch variants of any project can run directly in avs

DTL

30th March 2023, 06:19

"It uses 2 past frames to predict next frame"

It can be easily tested in tr=2 temporal denoising:
n-2 and n-1 feed as t-1 and t frames from 2 past frames forward interpolation,
n+2 and n+1 feed as pair of 2 next frames for backward interpolation to the past (engine should be time-axis symmetrical and do not know real time axis direction - sort of TENET movie idea)
got 2 interpolated frames from 2 previous and 2 next - and pass it after interleaving with current n-frame to blending engine like vsTTempSmooth (sample-based, or mvtools blocks-based).

Also developers promises finally move to multi-frames transforms analysis for better prediction and compensation for complex motion/transforms in case of non-constant speed motion and so on. But when it be released in some working demo may be still unknown.

As I understand from paper https://arxiv.org/pdf/2303.09875.pdf there is still very active scientific research on image processing exist in some asian region (China ?) but the results still far enough for real testing and/or usage in AVS. But also from that paper it looks that the research group completely miss the main important task for interpolation engines of temporal denoising and MPEG compressability improving. So the currently engines in development can not be directly used for replacement denoise engines with any-tr like mvtools/MDegrainN. And 'large and very large' tr of about 10 or even 100+.

Reel.Deel

4th April 2023, 00:54

Here's some other models compatible with avs-mlrt: https://github.com/the-database/mpv-upscale-2x_animejanai/

2x_AnimeJaNai is a set of realtime 2x Real-ESRGAN Compact, UltraCompact, and SuperUltraCompact models intended for high or medium quality 1080p anime to 4k with an emphasis on correcting the inherit blurriness of anime while preserving details and colors. These models are not suitable for artifact-heavy or highly compressed content as they will just sharpen artifacts. The models can also work with SD anime by running the models twice, first from SD to HD, and then HD to UHD.

They are already onnx models so I won't add them to the collection.

---

@dogway, I'll get to your requested models in a bit ... I've been away from my home PC.

anton_foy

4th April 2023, 08:08

Found https://huggingface.co/utnah/esrgan/tree/main

Maybe of interest?

FranceBB

19th May 2023, 13:44

Hi there!
I'm late to the party, but hey, never say never ehehehehe
So, I tried with:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(list_gpu=true)

and indeed it shows my NVIDIA GTX 980Ti

but when I tried with:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="\\avs000\Ingest\MEDIA\temp\onnx-models\VHS-Sharpen-1x_46000_G.onnx", builtin=false, list_gpu=false)

https://i.imgur.com/B3nvqIH.png

of course I have all the C++ Redistributable installed:

https://i.imgur.com/9BM8dWh.png

and, despite the error, I can see the GPU VRAM being used:

https://i.imgur.com/kfGawUx.png

I was running Avisynth 3.7.3 x64 Beta 9 by Ferenc Pinter.
I tried to switch to the IntelLLVM builds as suggested in the read-me on GitHub, but it didn't make any difference.

On the other hand, when I tried a different model, it worked.
For instance:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="\\avs000\Ingest\MEDIA\temp\onnx-models\1x_BroadcastToStudioLite_485k.onnx", builtin=false, list_gpu=false)

worked:

https://i.imgur.com/cWh0wTT.png

Before I fire up my Quadro P4000 and P5000, is it because the GTX 980Ti is too old for some models or is there some other reason behind it?

In particular, it looks like the models whose .onnx files are 65MB don't work and the ones that are smaller do.
For instance, 1x_ThePi7on-Solidd_Deborutify_UltraLite_260k_G.onnx also worked (and it's indeed just 4.6 MB).
Same goes for realesr-general-wdn-x4v3.onnx, which worked just fine (https://i.imgur.com/JcEYt7R.png) and again it's just 4.7MB.

Reel.Deel

19th May 2023, 15:05

My cheap GPU has trouble with the larger models. I can get the VHS-Sharpen-1x_46000_G model to work using the tilesize and overlap options and also fp16.

mlrt_ncnn(network_path=model, builtin=false, fp16=true, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)

-----

While I'm here, I found some other models in the following pages.

Large collection of onnx models: https://github.com/styler00dollar/VSGAN-tensorrt-docker/releases/tag/models

Phhofm models (some already onnx): https://github.com/Phhofm/models

FranceBB

19th May 2023, 16:41

Gotcha!
Yep, that way it worked, thanks! :)

FranceBB

25th May 2023, 14:16

I tested realesr-general-wdn-x4v3_opset16.onnx

Here are the results depicted in some nice previews, each with its SSIM score.

Below you can find the script:

#Indexing
LWLibavVideoSource("Test.mxf")
#ImageSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\Lenna_(test_image).png")
Bob().Spline64Resize(848, 480)
original=ConvertBits(8).Converttoyv12().Text("Original", y=66)

#Downscale
SinPowResizeMT(width/4, height/4)

#Various Upscales
point=PointResize(width*4, height*4).ConvertBits(8).Converttoyv12()
#bilinear=BilinearResize(width*4, height*4).Converttoyv12()
nnedi3=nnedi3_rpow2(cshift="Spline64ResizeMT", rfactor=2, fwidth=width*4, fheight=height*4, nsize=4, nns=4, qual=1, etype=0, pscrn=2, threads=0, csresize=true, mpeg2=true, threads_rs=0, logicalCores_rs=true, MaxPhysCore_rs=true, SetAffinity_rs=false).ConvertBits(8).Converttoyv12()
esrgan=last.ConverttoPlanarRGB().ConvertBits(32).mlrt_ncnn(network_path="\\myshare\Ingest\MEDIA\temp\realesr-general-wdn-x4v3_opset16.onnx", builtin=false, list_gpu=false, fp16=true).ConvertBits(8).Converttoyv12()

#SSIM
pnt=SSIM(original, point, "\\myshare\Ingest\MEDIA\temp\point3SSIM.csv", "\\myshare\Ingest\MEDIA\temp\point3SSIM.txt", lumimask=1, scaled=0).Text("PointResize", y=66)
nne=SSIM(original, nnedi3, "\\myshare\Ingest\MEDIA\temp\nnedi3SSIM.csv", "\\myshare\Ingest\MEDIA\temp\nnedi3SSIM.txt", lumimask=1, scaled=0).Text("NNEDI3", y=66)
esr=SSIM(original, esrgan, "\\myshare\Ingest\MEDIA\temp\esrganSSIM.csv", "\\myshare\Ingest\MEDIA\temp\esrganSSIM.txt", lumimask=1, scaled=0).Text("ESRGAN", y=66)

#Preview
a=StackHorizontal(original, pnt)
b=StackHorizontal(nne, esr)

StackVertical(a,b)

And here are the images stacked as:

Original - PointResize
NNEDI3 - ESRGAN

I'm gonna pick one just to show why I'm sticking with NNEDI3:

https://i.imgur.com/A1b2eL6.png

Images collection:

Img1 (https://i.imgur.com/W7SXavj.png) - Img2 (https://i.imgur.com/yaRuaPC.png) - Img3 (https://i.imgur.com/NNmITRR.png) - Img4 (https://i.imgur.com/aUZs6iN.png) - Img5 (https://i.imgur.com/LEMrB9c.png) - Img6 (https://i.imgur.com/GCBFRkW.png) - Img7 (https://i.imgur.com/sBeZpS8.png) - Img8 (https://i.imgur.com/qtnf2hp.png) - Img9 (https://i.imgur.com/RgzDP6M.png) - Img10 (https://i.imgur.com/zxOJ1O7.png) - Img11 (https://i.imgur.com/Wcedbil.png) - Img12 (https://i.imgur.com/5rXpyBG.png) - Img13 (https://i.imgur.com/74H0zwA.png) - Img14 (https://i.imgur.com/ovQi3HV.png) - Img15 (https://i.imgur.com/KdWAAWo.png) - Img16 (https://i.imgur.com/NC8rIRR.png)

I guess I'm gonna stick with NNEDI3 for a while longer...

Reel.Deel

25th May 2023, 22:19

The result of that model looks very artificial. Have you tried any other models? I'd be worried about temporal consistency also. For real world video, the proprietary models from Topaz are good. Unfortunately those onnx models are housed in a password protected zip file :(.

Emulgator

26th May 2023, 01:55

Delivering a good choice from 186GB trained models Topaz (with Proteus v3 until 3.0.6 at least)
still had the "ugly face syndrom" when guessing at maybe-face-content from less than 20 pixel size.
For Proteus v4 this is promised to improve.
Some more training on face guessing from such small pixel patches should do.
But: Topaz VEAI 3 had Avisynth support killed, and users report many hassles.
My paid updates ended with 3.0.6, for the time being I wasn't willing to spend again, so I stick to last Topaz 2.6.4.

23.10.2023 23:21 Just today I came across my post by chance and oops:
Sorry for being completely OT. How did that get here ?
Must have been tired a bit on 26.03.2023 03:01, will try to move that where it belongs.

anton_foy

22nd October 2023, 17:57

convertbits(32)
converttoplanarrgb()

mlrt_ncnn(network_path="C:\Program Files (x86)\AviSynth+\plugins64\ml_\models\1x-Film-Degrainer-1-000.onnx", fp16=true, builtin=false, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8, list_gpu=true)

This loads the model fine and many (other models aswell) but it does nothing, it shows the original untouched clip. How to set the strength?
Or is it something else Im missing?

kedautinh12

23rd October 2023, 00:39

You can delete tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8 if your GPU stronger than Reel.Deel's GPU

Reel.Deel

23rd October 2023, 01:50

@anton_foy

In your script you have list_gpu=true, try setting it to false. As for a strength setting, there is none.

anton_foy

23rd October 2023, 08:51

@anton_foy

In your script you have list_gpu=true, try setting it to false. As for a strength setting, there is none.

Thanks yes it works now! But really slow even with the RTX 4070.

kedautinh12

23rd October 2023, 10:05

Did you try delete my recommended parameters too?

anton_foy

23rd October 2023, 12:51

Did you try delete my recommended parameters too?

Thanks I will try this too. So by adjusting tilesize and overlap it can work for slower cards but slower processing rather than the default values for a faster card?

kedautinh12

23rd October 2023, 15:44

Yeah, I tried delete it and the plugin will faster cause it will use more GPU memory. Try change fp16 to false too. If you meet error, you can change back to true

Reel.Deel

23rd October 2023, 21:05

Thanks I will try this too. So by adjusting tilesize and overlap it can work for slower cards but slower processing rather than the default values for a faster card?

Using tilesize just means that the image is divided into sections, and because of overlap you end up processing more pixels. I don't think setting fp16 to true will have any negative effects on speed, even on higher end GPUs.

anton_foy

24th October 2023, 08:15

Thanks I tried both of your suggestions but it is impossibly slow and I tried Vapoursynth vs-scunet on the same clip and 30 frames would take over 1 hour. So I guess this processing UHD with these models is out of the question for now.

kedautinh12

24th October 2023, 08:58

Waiting for cuda ver
https://github.com/Asd-g/avs-mlrt/issues/7#issuecomment-1771187328

anton_foy

25th October 2023, 08:56

@kedautinh12
Thanks for the info good news.
Something I am wondering about is the num_streams param. Where to find the specification about this for my GPU? May this speed up if I put this to max for my card?

Selur

25th October 2023, 18:51

The number of stream depends on the resolution of your input and the VRAM that is used with model X and 1 stream.
If you use a model with one stream and only a fraction of your VRAM is used, you can run multiple streams in parallel to speed things up.
=> increasing the streams will cost VRAM, but can speed things up
So if either the input resolution or the used model changes, the VRAM usage and thus the amount of streams your card can handle can change.

anton_foy

26th October 2023, 07:46

kedautinh12

22nd November 2023, 01:49

Would like to see SwinIR models:
https://github.com/JingyunLiang/SwinIR/releases
https://openmodeldb.info/?q=SwinIR

Dogway

22nd November 2023, 02:43

Yes, me too. I tried in vapoursynth and SwinIR didn't work for me there either. Looking for a solid photo upscaler.

kedautinh12

22nd November 2023, 03:08

For Vapoursynth, why you don't ask HolyWu?
https://github.com/HolyWu/vs-swinir/issues?q=is%3Aopen+is%3Aissue

Selur

22nd November 2023, 05:42

https://github.com/HolyWu/vs-swinir works fine here,...

Dogway

22nd November 2023, 20:06

Ok, I thought the SwinIR models in pth format were supposed to work. Thanks both.

kedautinh12

1st December 2023, 14:51

Would like to see CodeFormer models :D
https://github.com/sczhou/CodeFormer/releases

Selur

2nd December 2023, 09:44

Assuming some folks here use those models on normal content, how do you deal with temporal inconsistencies?
(like this: https://forum.videohelp.com/attachments/75185-1701473592/Face.mp4)
Personally, I use QTGMC with input type 3 to lessen this, but I was wondering what others do.

Cu Selur

kedautinh12

3rd December 2023, 03:16

Would like to see UniScale models
https://openmodeldb.info/?q=Uniscale

Selur

3rd December 2023, 08:21

You can convert those with chaiNNer (https://github.com/chaiNNer-org/chaiNNer),... (these are also included in the Hybrid onnx models collection)

kedautinh12

20th December 2023, 04:27

Would like to see SRFormer models
https://openmodeldb.info/?q=SRFormer

Selur

20th December 2023, 08:05

Would like to see UniScale models
Those seem to work fine here with vs-mlrt. (they are rather slow)
Would like to see SRFormer models
Best post such request over in the github issue tracker.

Cu Selur

kedautinh12

20th December 2023, 10:07

I can see it in other model. They use srformer to base for trained on
https://github.com/Phhofm/models

kedautinh12

21st December 2023, 04:52

Would like to see Cutie models
https://github.com/hkchengrex/Cutie?tab=readme-ov-file

takla

19th February 2024, 10:38

Actually it runs "ok" as long as you're not using any of the broken models from the dropbox link (I haven't tested them all, but 4 of 4 gave an error)
Should be fine if converted manually with "opset 17"

takla

10th March 2024, 16:42

Here are some models (https://www.mediafire.com/file/pocrytgzzejlsan/models.zip/file) which I've converted to onnx (Mostly for Cartoons/Anime)
I can vouche for all of them working (tested on AMD GPU)
Original sources can be found here (https://openmodeldb.info/)

2x_AniScale2S_Compact_i8_60K
Very good with BluRay sources

2x-LD-Anime-Compact
Amazing with high quality DVDs

2x_Garfieldjr_span48
Very Good for mediocre DVD quality

Example script:
LWLibavVideoSource("INPUT.mkv")
ConvertBits(32).ConvertToPlanarRGB()
mlrt_ort(network_path="C:\Program Files (x86)\AviSynth+\plugins64+\models2\2x-LD-Anime-Compact.onnx", builtin=false, provider="dml")

DTL

21st April 2024, 13:48

Some ideas: If AI/NN plugins typically accept only RGB in float - do the current models/kernels make some benefit in quality if being fed with linear RGB instead of transfer-encoded ? The float samples format allows us to feed linear data without significant quantization distortions. And plugins like Avsresize allows to do both integer to float conversion and YUV decoding and also transfer function decoding to linear RGB in a single call.

Each model/kernel may be trained with unknown dataset for the samples domain encoding - either some system transfer-encoded (601/709/HDR) or linear.

poisondeathray

21st April 2024, 14:01

Some ideas: If AI/NN plugins typically accept only RGB in float - do the current models/kernels make some benefit in quality if being fed with linear RGB instead of transfer-encoded ?

Usually not, because 99.9% (probably 100%) of the models are trained on non linear data sets (and that includes 100's of other machine learning projects currently available but not as avs/vpy plugins) . Even commercial machine learning projects use gamma encoded sources

But you can train one on linearized data if you wanted to - I doubt it would provide any substantial benefit for current tasks

One area that would benefit from models trained on linearized datasets, is compositing. All high end compositing is done in 32bit linear environment. So inferencing/predictive models in that scenario could be useful

simple_simon

25th April 2024, 20:49

Are there any models that help with deblending bad pal-->ntsc conversions? The kind that QTGMC+srestore can't handle. Also any good ones for deblocking?

SaurusX

26th April 2024, 14:45

Are there any models that help with deblending bad pal-->ntsc conversions? The kind that QTGMC+srestore can't handle. Also any good ones for deblocking?

There really has not been much advancement in that area. The best option to use is still animeivtc(mode=2), whether it's actually anime or live action.

simple_simon

26th April 2024, 20:31

It seems like a perfect use case for AI. Train it to recognize what a blend looks like (even outside of a regular pattern) and deblend just those frames.

Selur

26th April 2024, 21:26

Probably, but (nearly) all models in vs/avs-mlrt have no temporal component and only filter one image at a time (without taking the surrounding frames into account), I doubt an algorithm can decide on a single image on how the deblended version should look like.

DTL

26th April 2024, 23:10

Any good denoiser for moving pictures must be temporal. Several GBs GPU board can fit many frames for temporal processing.

Even 'simple' RIFE make processing based on 2 input frames. And we expect next RIFE will use more frames (to estimate not only 2-points speed but several points acceleration for better motion interpolation).

Selur

27th April 2024, 13:52

@DTL: So, your conclusion is that that most of the models/filters that can be used with vs-/avs-mlrt can't be good since they only work on a per frame bases?

DTL

27th April 2024, 15:45

It may be good if filter is not depends on data in the frame. Like ColorYUV or Levels. But if it depends on frame data - processing of several frames in a sequence may cause additional temporal non-stability.
But typical nature of NN/AI filters is very deep dependence on the data to process. So better to ask NN/AI engine developers to make multi-frame engines for better temporal stability.

kedautinh12

27th April 2024, 16:16

DTL

28th April 2024, 08:20

HD/UHD frames are only MBs in size. Not GBs. So I think it is only still poor optimization of AI/NN for moving pictures processing at current hardware. As I see the main task for many current NN/AI developers is only make some working demo for 1 frame processing (for publication and level-up) but not working product as freeware. So they put close to zero efforts for optimizing for movies processing. As example of good marketing product I see Topaz VEAI - it works very good at typical workstation and very fast (in comparison wih may other freeware solutions).

tormento

21st October 2024, 19:08

Starting now to understand mlrt and my first script fails :)

LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\DehaloAlpha\Dehalo_alpha.avsi")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\Dither\mt_xxpand_multi.avsi")
Import("D:\Eseguibili\Media\StaxRip\Apps\Plugins\AVS\FineDehalo\FineDehalo.avsi")
DGSource("M:\In\Monster ~R2J\1-01-B5.dgi",ct=4,cb=0,cl=4,cr=4)
DGTelecide(mode=1, pthresh=3.5)
DGDecimate()
z_convertformat(pixel_type="RGBPS", use_props=0)
mlrt_W2x(backend="ort", model=1, noise=-1, scale=2)

gives me the error:

I don't know what 'height_' means.
(D:/Programmi/Media/AviSynth+/plugins64+/mlrt_ort-1.1.5-AsdG.avsi, line 203)
(D:/Programmi/Media/AviSynth+/plugins64+/mlrt_ort-1.1.5-AsdG.avsi, line 206)
(M:\In\Monster ~R2J\mlrt_waifu.avs, line 13)

Any help is welcome

StainlessS

21st October 2024, 20:02

Maybe take a look at mlrt_ort-1.1.5-AsdG.avsi, line 203. [find "height_"]
Show the full error line 203 from above.
height_ is not defined.

tormento

21st October 2024, 22:57

Show the full error line 203 from above.
height_ is not defined.
I already looked at it. In that line there is a call to that variable and it's not probably been declared yet.

The question is if somebody else has that script working (I simply renamed it) or if it's a bug.

Arx1meD

31st October 2024, 17:02

When using mlrt_ncnn, some ONNX models give an error:
System Exception - Access Violation
What can give such error?

DTL

31st October 2024, 21:18

Try different drivers for GPU and lower settings if possible (like lower threads and all possible other adjustable resources). Or may be GPU RAM size too low.

Arx1meD

1st November 2024, 08:37

I have a NVIDIA RTX 2060 Super, RAM - 8 GB.
I noticed that only ESRGAN and Compact architecture models work. This is written in the notes to the chaiNNer program - readme-ov-file # onnx (https://github.com/chaiNNer-org/chaiNNer?tab=readme-ov-file#onnx). Probably this could be the cause of the error.

kedautinh12

1st November 2024, 17:06

Arx1meD

1st November 2024, 18:31

I process DVD and video with a frame size 640x480.

FranceBB

1st November 2024, 20:18

I process DVD and video with a frame size 640x480.

Does it also happen with fp16=true?
I also have 8GB GDDR5 in my NVIDIA Quadro P4000 and I had to add that to compensate 'cause otherwise it would perform calculations in 32bit float and run out of memory.

kedautinh12

2nd November 2024, 05:19

Does it also happen with fp16=true?
I also have 8GB GDDR5 in my NVIDIA Quadro P4000 and I had to add that to compensate 'cause otherwise it would perform calculations in 32bit float and run out of memory.

Yes, the 32bit is very strong for even with 8gb vram GPU in 480p video

Arx1meD

2nd November 2024, 06:15

Does it also happen with fp16=true?
Yes. When fp16=true also appears errors.
I also noticed when the models are converted in the chaiNNer program with the selection fp32 everything works fine (for ESRGAN architecture models), and appears errors for the same model with the selection fp16.

Stereodude

17th February 2025, 17:54

FWIW, mlrt_ort using CUDA is faster than mlrt_ncnn to run an ONNX model (Dotzilla) on a RTX 3070Ti using a DVD source. I think it was 24fps vs. 18fps.

takla

15th September 2025, 15:13

Posting this example script here, so I don't forget to set colorspace stuff again...

Also, Cuda is waay slower for me, barely reaching 1/3 of the speed I get with DML.
This has also been shown by other to be the case (https://github.com/AmusementClub/vs-mlrt/releases/tag/v13.2).

With Hardware-accelerated GPU scheduling enabled, GPU usage often gets stuck at 80%.
If you run into this issue, you can disable it under Windows settings "System > Display > Graphics >> Advanced graphic settings"
Edit: Someone else (https://github.com/AmusementClub/vs-mlrt/discussions/19#discussioncomment-4750372) ran into the same issue.

LWLibavVideoSource("C:\Users\ULTRA\Downloads\ffmpeg\INPUT.mkv").Prefetch(0)
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:limited=>rgb:709:709:full", dither_type="error_diffusion", cpu_type="avx2", use_props=0).Prefetch(0)
mlrt_ort(network_path="C:\Program Files (x86)\AviSynth+\plugins64+\mlrt_ort_rt\models\2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx", builtin=False, provider="DML", fp16=True, num_streams=1).Prefetch(2)
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited", dither_type="error_diffusion", cpu_type="avx2", use_props=0).Prefetch(0)

takla

5th February 2026, 23:51

Exciting news:
ONNX is coming to FFMPEG (https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21532)

DTL

6th February 2026, 09:01

A question to NN/mlrt filters developers:

Currently RIFE uses internal float32 range remap to x255.0f (0..1 to 0..255) and all other (3) filters form avs-mlrt uses direct data upload from AVS to accelerator (in its range of 0..1).
Is it a special feature required by RIFE models/processing ? Can we use direct AVS 0..1 float range data with RIFE without issues ? Or RIFE models were specifically learned on 0..255 old PC BMP data and can not process 0..1 data range with same quality ?

The idea is to add new filters interconnection mode to AVS like GetFrameToBuffer() to direct data write into allocated upload buffer to accelerator instead of large RGBPS data copy from AVS cache to allocated upload buffer by CPU.

WolframRhodium

6th February 2026, 11:12

Currently RIFE uses internal float32 range remap to x255.0f (0..1 to 0..255)

What do you mean by that? It uses (https://github.com/Asd-g/avs-mlrt/blob/85856d174d072ef97ae1986184b4776ea50b4849/scripts/mlrt_RIFE.avsi#L155) 0..1 float range.

The idea is to add new filters interconnection mode to AVS like GetFrameToBuffer() to direct data write into allocated upload buffer to accelerator instead of large RGBPS data copy from AVS cache to allocated upload buffer by CPU.

I am not a fan of this. We can hide the latency of data transfer if it is shorter than the computation time, without changing existing software stack. (Of course, this could be necessary for lightweight models, but I don't think it is significant in the near future.)

DTL

6th February 2026, 11:49

What do you mean by that? It uses (https://github.com/Asd-g/avs-mlrt/blob/85856d174d072ef97ae1986184b4776ea50b4849/scripts/mlrt_RIFE.avsi#L155) 0..1 float range.

At input it accepts 'standard' AVS float32 range of about 0..1 but at making copy into allocated by accelerator library buffer it makes additional range remap to 0..255 - https://github.com/Asd-g/AviSynthPlu.../rife.cpp#L379
in0R[w * y + x] = src0R[src_stride * y + x] * 255.0f;
in0G[w * y + x] = src0G[src_stride * y + x] * 255.0f;
in0B[w * y + x] = src0B[src_stride * y + x] * 255.0f;
in1R[w * y + x] = src1R[src_stride * y + x] * 255.0f;
in1G[w * y + x] = src1G[src_stride * y + x] * 255.0f;
in1B[w * y + x] = src1B[src_stride * y + x] * 255.0f;

The all other (3) avs-mlrt plugins to make a copy to upload buffer simply call avs_bitblt (copy without range remap).

The question: is it some obligatory feature of RIFE models or some old redundant operation and can be removed (if we can write directly to the upload buffer with 'standard' AVS float32 range mapping about 0..1) ?

"We can hide the latency of data transfer if it is shorter than the computation time, "

The idea is to free CPU and host memory bus resources. While we making some AVS filters processing we also make MPEG encoding with x26x software and it is also very slow. If we free some CPU + memory bus resources it is expected to have some performance boost with total processing of AVS+MPEG encoding (onCPU). Copy of the largest RGBPS format is very bad for cache and memory bus load.
So it is planned to test if we can simply make uncached stores at the output of
ConvertToPlanarRGB(bits=32) new filter in AVS+ core directly in the upload buffer of accelerator.
It is the special type of filters without CPU-core-readback for processing. It is filters-adapters to feed data to external accelerator (read from host RAM via DMA ?).

Updated plugin expected like:

ncnn::Mat in0;
ncnn::Mat in1;
in0.create(w, h, channels, sizeof(float), 1); // allocate upload buffers to accelerator
in1.create(w, h, channels, sizeof(float), 1);
float* in0R{ in0.channel(0) };
float* in0G{ in0.channel(1) };
float* in0B{ in0.channel(2) };
float* in1R{ in1.channel(0) };
float* in1G{ in1.channel(1) };
float* in1B{ in1.channel(2) };

child->GetFrameToBuffer(frame_num, buffer_description(in0R,...)); // uncached write from CPU core
to upload buffers for DMA to accelerator in large RGBPS format

ncnn::VkCompute cmd(vkdev);

StvG

6th February 2026, 13:08

At input it accepts 'standard' AVS float32 range of about 0..1 but at making copy into allocated by accelerator library buffer it makes additional range remap to 0..255 - https://github.com/Asd-g/AviSynthPlu.../rife.cpp#L379...

@DTL, AviSynthPlus-RIFE - https://github.com/Asd-g/AviSynthPlus-RIFE/blob/main/src/rife.cpp is completely different plugin than avs-mlrt - https://github.com/Asd-g/avs-mlrt/blob/85856d174d072ef97ae1986184b4776ea50b4849/scripts/mlrt_RIFE.avsi

WolframRhodium

6th February 2026, 14:19

The idea is to free CPU and host memory bus resources.

There are tons of similar improvements that could make a massive difference, e.g. fusing pointwise operations like MakeDiff() with resize. And PCIe 5.0 is fast.

tormento

7th February 2026, 14:38

And PCIe 5.0 is fast.
Remember that somebody hasn't the money to upgrade to a modern system. :(

WolframRhodium

7th February 2026, 16:42

Remember that somebody hasn't the money to upgrade to a modern system. :(

Then similarly the gpu processing time will be longer than the data transfer time.

tormento

7th February 2026, 16:44

Then similarly the gpu processing time will be longer than the data transfer time.
Did you use all the possible tricks on newer builds of AVS+ to decrease the overhead from GPU to CPU transfers to have the temporal part processed?

DTL

7th February 2026, 23:26

If host CPU load will be lower - user can set higher quality MPEG encoding params and get more quality at the same encoding time (limited by accelerator performance).

Currently no significant optimizations for data transfer from host memory to GPU in latest AVS+ builds. The only optimization in r4483 build is faster YUV to RGBPS transform with single YUV data load to CPU instead of separate ConvertBits(32) load/store stage.

Now users can call single ConvertToPlanarRGB(bits=32) and it is faster in comparison with old sequence of
ConvertToPlanarRGB()
ConvertBits(32)
and also may be a bit higher in precision (in case of integer YUV dematrix before ConvertBits(32)).