Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th February 2023, 03:43   #1  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
ONNX models to use with avs-mlrt

PyTorch models converted to ONNX to use with AviSynth+'s avs-mlrt plugin. Models were converted with chaiNNer (fp32 data type). Let me know of missing models to convert and I'll add them to the package.


Download: onnx-models-rd

Other repositories that include onnx models:
Model database
Basic usage
Code:
FFVideoSource("YUV420Source.mkv")
ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="1x_Dotzilla_Compact_80k_net_g.onnx", builtin=false)
You can omit the builtin parameter if the models are in the same location as the avs-mlrt plugin (see builtin and builtindir parameters in the docs).

If you have a low end GPU and trying to use the larger models, it helps to use the tilesize, overlap and fp16 options.
Code:
mlrt_ncnn(network_path=model, builtin=false, fp16=true, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)

Model conversion

Download chaiNNer and install the required dependencies. Load the corresponding nodes, connect them and then hit run by either pressing the play button at the top or F5.


Last edited by Reel.Deel; 25th May 2023 at 21:57. Reason: add links and restructure
Reel.Deel is offline   Reply With Quote
Old 26th February 2023, 03:57   #2  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Code:
Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 26th February 2023, 04:02   #3  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
wow, so i think this will open new era about pytorch can port to avisynth
kedautinh12 is offline   Reply With Quote
Old 26th February 2023, 04:03   #4  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
Quote:
Originally Posted by Dogway View Post
Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Code:
Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate
Can it use for real world video??
kedautinh12 is offline   Reply With Quote
Old 26th February 2023, 05:06   #5  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by Dogway View Post
Woow, so great, ''thumbs up'', these are my go to for anime and manga. Still searching a good one for photos though.

Code:
Old Anime Upscale:
    1x_Dotzilla_Compact_80k_net_g.pth (dot crawl and rainbowing)
    2X_DigitalFilmV5_Lite.pth (sharpener for soft lines, no need to downscale before AI)
    2x_AnimeClassics_UltraLite_510K.pth (less reconstruction than above, similar to Waifu but for Anime. Color Shift)
    2x_LD-Anime_Skr_v1.0.pth (for ringing, rainbowing, aliasing)
Manga Upscale (B/W):
    4x_eula_digimanga_bw_v1_860k.pth
Magazines?
    4x-UniScaleV2_Moderate
Thanks for the suggestions. I added them to the package. I'll provide a Dropbox link from now on since the size is already at 331mb. This way people can pick and choose only the models they want. I did not test these new models, let's hope they don't blow up

Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).

Last edited by Reel.Deel; 27th February 2023 at 08:45. Reason: Delete link, see 1st post
Reel.Deel is offline   Reply With Quote
Old 26th February 2023, 05:17   #6  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by kedautinh12 View Post
Can it use for real world video??
Usually the models are quite narrow in scope; typically they are only useful on the data set that they were trained on, or very similar material. But you can try it...



Quote:
Originally Posted by Reel.Deel View Post
Edit: if anyone has the "DeGif" model please share it. The link to it is not working right now (Public Transfer Limit Exceeded).
Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 05:19   #7  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by poisondeathray View Post
Which version ? There are older checkpoint versions and the most recent version I have is "2x_NMKD-DeGIF_210000_G.pth"
I guess the latest one. There is no version number on the upscale wiki.
Reel.Deel is offline   Reply With Quote
Old 26th February 2023, 05:22   #8  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 05:33   #9  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by poisondeathray View Post
I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?
Thanks for the models.

1) Depends on the size of the model. Small ones are quick (few seconds), the largest one I converted (64mb) took around a minute.

2) I have not compared them to the original version. It would be interesting to do so. But while researching how to convert between the two formats I only saw that mentioned once or twice. See here: https://www.reddit.com/r/MachineLear...model_to_onnx/ and here https://pytorch.org/docs/stable/onnx.html#limitations

I'm all ears if anyone has any suggestions.

This is the interface in ChaiNNer:

I'm sure there more options when using PyTorch directly but I have not experimented with that.

Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.

Last edited by Reel.Deel; 26th February 2023 at 07:29.
Reel.Deel is offline   Reply With Quote
Old 26th February 2023, 09:00   #10  |  Link
Arx1meD
Registered User
 
Arx1meD's Avatar
 
Join Date: Feb 2021
Posts: 121
This plugin works on GPU only?
Arx1meD is offline   Reply With Quote
Old 26th February 2023, 11:56   #11  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
For denoise process (in physical temporal dimension) we need models for motion compensation. The framerate interpolation models are only partially good for this processing. Better neural network engines for motion compensation need accept several input frames in a sequence (not only 2 frames as we have in RIFE now).

The 'training condition' for motion compensation process is simple enough: For given number of frames with marked 'current' frame it is required to create 'current' frame view using only 'other' frames. In simplest form with 2 frame only input - we have only 'current' and 'other' frame. So engine need to reconstruct 'current' frame using only samples data from 'other' frame (it can use 'current' frame only as 'objects positions reference').

It may be even more simple in compare with 'framerate conversion' process where engine have 2 frames 'before and after' and 'current' frame to reconstruct at time of model training.

Quote:
Originally Posted by kedautinh12 View Post
Can it use for real world video??
One issue I found when trying to use online AI upscalers - they may not understand the depth of field blur and start to attempt to make out of focus objects sharper too. So for simple 2D drawing without textures like anime it work for sharp lines restoration from aliasing and all frame deblur about good. But for 3D real world lens projection to 2D buffer + different type (and DOF blur) textures it may fail significantly.

Last edited by DTL; 26th February 2023 at 12:11.
DTL is offline   Reply With Quote
Old 26th February 2023, 15:34   #12  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Quote:
Originally Posted by Reel.Deel View Post
Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.
Didn't crash for me, but the GPU got high rolling for a few seconds... scared.

Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 26th February 2023 at 16:36.
Dogway is offline   Reply With Quote
Old 26th February 2023, 16:10   #13  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
DTL: The 'training condition' for motion compensation process is simple enough: For given number of frames with marked 'current' frame it is required to create 'current' frame view using only 'other' frames. In simplest form with 2 frame only input - we have only 'current' and 'other' frame. So engine need to reconstruct 'current' frame using only samples data from 'other' frame (it can use 'current' frame only as 'objects positions reference').

It may be even more simple in compare with 'framerate conversion' process where engine have 2 frames 'before and after' and 'current' frame to reconstruct at time of model training.
Tried to find any model for this kind of motion compensation but without success. One would think there would be some out there? Although I do not specifically know what words to use for searching. I tried "model pytorch motion compensation" but mostly got hits about object tracking and similar.

Edit: if someone knows how to do the training bit I can provide with before/after tests in UHD (Sony does 6k then downscales internally to 3840x2160 and it is so crisp with minimal halo when internal sharpness is turned off).

Last edited by anton_foy; 26th February 2023 at 16:15.
anton_foy is offline   Reply With Quote
Old 26th February 2023, 17:14   #14  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Dogway View Post
Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.
There are some inpainting algos that use different architectures (cannot be run in avs/vpy/cupscale)

https://github.com/geekyutao/Image-Inpainting
https://github.com/zengyh1900/Awesome-Image-Inpainting

The problem I found with single image/spatial inpainting algos, is the flicker and temporal inconsistencies (you'd have to apply some temporal smoothing after)

Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo


Chainner lists support for 2 spatial algos that I haven't tried yet; but they should "work" since the GUI has direct support

Lama
https://github.com/advimman/lama

MAT
https://github.com/fenglinglwb/MAT





Quote:
Originally Posted by anton_foy View Post
Tried to find any model for this kind of motion compensation but without success. One would think there would be some out there? Although I do not specifically know what words to use for searching. I tried "model pytorch motion compensation" but mostly got hits about object tracking and similar.
Object tracking is part of it. Also propogation and alignment.

There are typically catergorized into single image vs. video . "Video" algorithms takes >1 frame into account. All the ESRGAN / Real-ESRGAN derivatives are single image only.

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

I posted some examples of BasicVSR/BasicVSR++ in a few threads here and other forums. The provided model works on "textbook" clean cases. A big problem for is degradation step wasn't included , so it's not as generalizable to real world, compression cases. RealBasicVSR adds degradation, but the results are quite blurry / detail loss
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 17:22   #15  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Reel.Deel View Post
Edit: can someone test the 2x_LD-Anime_Skr_v1.0.pth model? It crashes on me.
Could be GPU memory ?

Did you try the tiling options ?

Quote:
- overlap_w, overlap_h\
Overlap width and overlap height of the tiles, respectively.\
Must be less than or equal to `tilesize_w` / `tilesize_h` `/` `2`.\
Default: 0.

- tilesize_w, tilesize_h\
Tile width and height, respectively.\
Use smaller value to reduce GPU memory usage.\
Must be specified when `overlap_w` / `overlap_h` > 0.\
Default: input_width, input_height.

And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 17:34   #16  |  Link
DonCanjas
Registered User
 
DonCanjas's Avatar
 
Join Date: Jul 2022
Posts: 4
Would like to see
  • 4x_eula_digimanga_bw_v2_nc1_307k
  • MangaJPEG
  • 1x_GainRESV3 (Aggro,Natural,Passive)
  • 1x_DitherDeleter-Smooth_104000_G
__________________
Check out my repo for stupid and/or possibly useful avs scripts: https://github.com/DonCanjas/Avisynth-Functions
DonCanjas is offline   Reply With Quote
Old 26th February 2023, 17:48   #17  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
@VX will happy if had delogo models work perfectly
kedautinh12 is offline   Reply With Quote
Old 26th February 2023, 17:57   #18  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by poisondeathray View Post
And I tried a few ONNX converted models , they were bit identical to the pytorch version so far...
Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 18:01   #19  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,346
Quote:
Originally Posted by Arx1meD View Post
This plugin works on GPU only?
I think so, the requirements say Vulkan device. I do not see a "CPU" mode anywhere

https://github.com/Asd-g/avs-mlrt
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 18:05   #20  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Quote:
Originally Posted by poisondeathray View Post
Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries
And what is the current RIFE is based at ? Or it is separated engine for doing motion interpolation ? Some wiki https://www.svp-team.com/wiki/RIFE_AI_interpolation says it is based on IFNet neural network.
Also it shows TensorRT at NVIDIA cards can run RIFE about 2x faster in compare with ncnn/Vulkan implementation. So RIFE (IFNet ?) can be compiled (?) to different hardware APIs ?

As for only storing frames in memory - the 1920x1080 frame in 12bytes per sample RGB 32bit float is about 24 MB only. So 1/4 of 1GB onboard RAM can keep 10 frames and it is very good already to study how motion is going between several frames.

I post an issue on some RIFE development project at github about possible changing RIFE algorithm for better denoise process - https://github.com/megvii-research/E...IFE/issues/304 but still no one reply. May be it require to search someone with understanding how RIFE work so the required changes can be implemented.

Last edited by DTL; 26th February 2023 at 18:13.
DTL is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:51.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.