ONNX models to use with avs-mlrt

[Reel.Deel]

"o" and "last" have different dimensions.

My bad, I completely overlooked that. :stupid:

But there might be something else going on. Interleave always throws an error when the dimensions are different. Can anyone else reproduce this behavior?

[poisondeathray]

But there might be something else going on. Interleave always throws an error when the dimensions are different. Can anyone else reproduce this behavior?


Just with a "normal" script ?


a=colorbars()
b=a.bicubicresize(a.width*2, a.height*2)

interleave(a,b)



Interleave: videos must be of the same size.

[Reel.Deel]

Just with a "normal" script ?


With a "normal" script it works fine. But when avs-mlrt is involved and the interleaved videos are different dimensions, it causes VDub2 to hang. At least for me. Can you try this script:

ColorBars(pixel_type="RGBPS")
o = last
mlrt_ncnn("2x_AnimeClassics_UltraLite_510K.onnx", builtin=false)
Interleave(o,last)
ConvertBits(8)

It works fine if o = last.PointResize(width*2, height*2).

[poisondeathray]

Can you try this script:

ColorBars(pixel_type="RGBPS")
o = last
mlrt_ncnn("2x_AnimeClassics_UltraLite_510K.onnx", builtin=false)
Interleave(o,last)
ConvertBits(8)



I get the "Interleave: videos must be of the same size. " message in avspmod; but vdub2 hangs

[Reel.Deel]

I get the "Interleave: videos must be of the same size. " message in avspmod; but vdub2 hangs

Same. I guess it's a VDub2 issue.

[takla]

Fails silently with my AMD Radeon GPU...
Standalone vulkan ncnn converters do work. So the culprit must be the avisynth plugin.

[poisondeathray]

Fails silently with my AMD Radeon GPU...
Standalone vulkan ncnn converters do work. So the culprit must be the avisynth plugin.

Did you fulfill the requirements in the ReadMe ?

Other common "culprits" for some people's silent fails were non meeting the avs+ minimum version, and GPU memory (use tiles) . Neither current produces a useful error message, which could be an improvement

[takla]

Did you fulfill the requirements in the ReadMe ?

Other common "culprits" for some people's silent fails were non meeting the avs+ minimum version, and GPU memory (use tiles) . Neither current produces a useful error message, which could be an improvement

Lmao. I literally updated everything but avisynth+. That was the issue. Works now with r3958. Thank you ;)

[Dogway]

I found new interesting models, not sure how feasible is it to convert to onnx, compatibility-wise:
Photos:
Real ESRGAN 4x Plus
Real ESRGAN General WDN 4x V3 (photos, paintings or anime illustrations)
Latent Diffusion Super Resolution (LDSR) (SUPER slow though)
SwinIR
Swin2SR
Remacri
Anime/LineArt Illustration Upscale:
Real ESRGAN 4x Plus Anime6B

Also sometimes when refreshing the filter in AvsPmod, all I get is a pink screen. Looks like some out of memory issues, even if I use tiles. Need to restart AvsPmod.

[Reel.Deel]

I found new interesting models, not sure how feasible is it to convert to onnx, compatibility-wise:
Photos:
Real ESRGAN 4x Plus
Real ESRGAN General WDN 4x V3 (photos, paintings or anime illustrations)
Latent Diffusion Super Resolution (LDSR) (SUPER slow though)
SwinIR
Swin2SR
Remacri
Anime/LineArt Illustration Upscale:
Real ESRGAN 4x Plus Anime6B


Do you have a link to Real ESRGAN 4x Plus and LDSR pth models?

I did not test them, but so far these models convert to to onnx without any problems:


realesr-general-wdn-x4v3.pth : https://github.com/xinntao/Real-ESRGAN/releases/tag/v0.3.0

RealESRGAN_x4plus_anime_6B.pth : https://github.com/xinntao/Real-ESRGAN/blob/master/docs/anime_model.md

Remacri : https://upscale.wiki/wiki/Model_Database#Universal_Models

SwinIR : https://github.com/JingyunLiang/SwinIR/releases

What specific SwinIR model you want converted, or all?

With the Swin2SR models (https://github.com/mv-lab/swin2sr/releases) I get an error when trying to covert:
An error occurred in a Convert To ONNX node:

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument tensors in method wrapper___cat)

Input values (partial):
• Data Type: fp32

Not sure what that means.

[Dogway]

Most models are posted here (https://github.com/xinntao/Real-ESRGAN/blob/master/docs/model_zoo.md).
Interested on realesrgan-x4minus as well, but the link is broken on the model_data page, I found a link in a reddit post, searching again now.

These are the Swin2SR models, the recommended for compressed jpeg photos. Haven't tested them, but as I could see Jpeg_dynamic seems the best, probably the others are also worth having a look:
Swin2SR_Jpeg_dynamic.pth
Swin2SR_ClassicalSR_X2_64.pth
Swin2SR_ClassicalSR_X4_64.pth
Swin2SR_CompressedSR_X4_48.pth
Swin2SR_Lightweight_X2_64.pth

Here (https://phhofm.github.io/upscale/favorites.html) they recommend SwinIR-L with CodeFormer (https://github.com/sczhou/CodeFormer#quick-inference) (faces) or GFPGAN (https://github.com/TencentARC/GFPGAN#european_castle-model-zoo) and v1.4 (https://github.com/chaiNNer-org/chaiNNer#face-restoration) (also faces), might be worth to check out.


About LDSR I found a post from here (https://www.reddit.com/r/StableDiffusion/comments/yev37i/comparison_of_upscaling_models_for_ai_generated/)that says:
chaiNNer does not support LDSR, but you can use it for example on replicate:
https://replicate.com/nightmareai/latent-sr

EDIT: You can find 4xLDSR now here (https://github.com/Phhofm/models/tree/main/4xLSDIRCompact/Version3) as onnx. The C version is good for heavy blocking images.

By the way, I wasn't aware either, but the wiki has also a page for "official" models which includes many of the ones listed above: https://upscale.wiki/wiki/Official_Research_Models

[DTL]

There is some promising update from neural-networks designers (authors of RIFE) about 'frame prediction' - https://github.com/megvii-research/CVPR2023-DMVFN . And recommended to check. It is expected to be better motion compensation engine in compare with current RIFE used in temporal denoising.

Can it be used in AVS via existing plugin or require plugin redesign ?

[poisondeathray]

There is some promising update from neural-networks designers (authors of RIFE) about 'frame prediction' - https://github.com/megvii-research/CVPR2023-DMVFN . And recommended to check. It is expected to be better motion compensation engine in compare with current RIFE used in temporal denoising.


It uses 2 past frames to predict next frame

I posted some examples in this thread.

https://forum.doom9.org/showthread.php?t=184387



Can it be used in AVS via existing plugin or require plugin redesign ?
Not currently.

If someone makes a ncnn/vulkan compatible version then possibly avs version could materialize. None of the direct pytorch variants of any project can run directly in avs

[DTL]

"It uses 2 past frames to predict next frame"

It can be easily tested in tr=2 temporal denoising:
n-2 and n-1 feed as t-1 and t frames from 2 past frames forward interpolation,
n+2 and n+1 feed as pair of 2 next frames for backward interpolation to the past (engine should be time-axis symmetrical and do not know real time axis direction - sort of TENET movie idea)
got 2 interpolated frames from 2 previous and 2 next - and pass it after interleaving with current n-frame to blending engine like vsTTempSmooth (sample-based, or mvtools blocks-based).

Also developers promises finally move to multi-frames transforms analysis for better prediction and compensation for complex motion/transforms in case of non-constant speed motion and so on. But when it be released in some working demo may be still unknown.

As I understand from paper https://arxiv.org/pdf/2303.09875.pdf there is still very active scientific research on image processing exist in some asian region (China ?) but the results still far enough for real testing and/or usage in AVS. But also from that paper it looks that the research group completely miss the main important task for interpolation engines of temporal denoising and MPEG compressability improving. So the currently engines in development can not be directly used for replacement denoise engines with any-tr like mvtools/MDegrainN. And 'large and very large' tr of about 10 or even 100+.

[Reel.Deel]

Here's some other models compatible with avs-mlrt: https://github.com/the-database/mpv-upscale-2x_animejanai/

2x_AnimeJaNai is a set of realtime 2x Real-ESRGAN Compact, UltraCompact, and SuperUltraCompact models intended for high or medium quality 1080p anime to 4k with an emphasis on correcting the inherit blurriness of anime while preserving details and colors. These models are not suitable for artifact-heavy or highly compressed content as they will just sharpen artifacts. The models can also work with SD anime by running the models twice, first from SD to HD, and then HD to UHD.

They are already onnx models so I won't add them to the collection.

---

@dogway, I'll get to your requested models in a bit ... I've been away from my home PC.

[anton_foy]

Found https://huggingface.co/utnah/esrgan/tree/main

Maybe of interest?

[FranceBB]

Hi there!
I'm late to the party, but hey, never say never ehehehehe
So, I tried with:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(list_gpu=true)

and indeed it shows my NVIDIA GTX 980Ti

but when I tried with:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="\\avs000\Ingest\MEDIA\temp\onnx-models\VHS-Sharpen-1x_46000_G.onnx", builtin=false, list_gpu=false)

https://i.imgur.com/B3nvqIH.png

of course I have all the C++ Redistributable installed:

https://i.imgur.com/9BM8dWh.png

and, despite the error, I can see the GPU VRAM being used:

https://i.imgur.com/kfGawUx.png


I was running Avisynth 3.7.3 x64 Beta 9 by Ferenc Pinter.
I tried to switch to the IntelLLVM builds as suggested in the read-me on GitHub, but it didn't make any difference.


On the other hand, when I tried a different model, it worked.
For instance:

ColorBars(848, 480, pixel_type="YV12")

ConvertBits(32)
ConvertToPlanarRGB()
mlrt_ncnn(network_path="\\avs000\Ingest\MEDIA\temp\onnx-models\1x_BroadcastToStudioLite_485k.onnx", builtin=false, list_gpu=false)

worked:

https://i.imgur.com/cWh0wTT.png


Before I fire up my Quadro P4000 and P5000, is it because the GTX 980Ti is too old for some models or is there some other reason behind it?

In particular, it looks like the models whose .onnx files are 65MB don't work and the ones that are smaller do.
For instance, 1x_ThePi7on-Solidd_Deborutify_UltraLite_260k_G.onnx also worked (and it's indeed just 4.6 MB).
Same goes for realesr-general-wdn-x4v3.onnx, which worked just fine (https://i.imgur.com/JcEYt7R.png) and again it's just 4.7MB.

[Reel.Deel]

My cheap GPU has trouble with the larger models. I can get the VHS-Sharpen-1x_46000_G model to work using the tilesize and overlap options and also fp16.

mlrt_ncnn(network_path=model, builtin=false, fp16=true, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8)


-----

While I'm here, I found some other models in the following pages.

Large collection of onnx models: https://github.com/styler00dollar/VSGAN-tensorrt-docker/releases/tag/models

Phhofm models (some already onnx): https://github.com/Phhofm/models

[FranceBB]

Gotcha!
Yep, that way it worked, thanks! :)

[FranceBB]

I tested realesr-general-wdn-x4v3_opset16.onnx

Here are the results depicted in some nice previews, each with its SSIM score.

Below you can find the script:


#Indexing
LWLibavVideoSource("Test.mxf")
#ImageSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\Lenna_(test_image).png")
Bob().Spline64Resize(848, 480)
original=ConvertBits(8).Converttoyv12().Text("Original", y=66)

#Downscale
SinPowResizeMT(width/4, height/4)

#Various Upscales
point=PointResize(width*4, height*4).ConvertBits(8).Converttoyv12()
#bilinear=BilinearResize(width*4, height*4).Converttoyv12()
nnedi3=nnedi3_rpow2(cshift="Spline64ResizeMT", rfactor=2, fwidth=width*4, fheight=height*4, nsize=4, nns=4, qual=1, etype=0, pscrn=2, threads=0, csresize=true, mpeg2=true, threads_rs=0, logicalCores_rs=true, MaxPhysCore_rs=true, SetAffinity_rs=false).ConvertBits(8).Converttoyv12()
esrgan=last.ConverttoPlanarRGB().ConvertBits(32).mlrt_ncnn(network_path="\\myshare\Ingest\MEDIA\temp\realesr-general-wdn-x4v3_opset16.onnx", builtin=false, list_gpu=false, fp16=true).ConvertBits(8).Converttoyv12()

#SSIM
pnt=SSIM(original, point, "\\myshare\Ingest\MEDIA\temp\point3SSIM.csv", "\\myshare\Ingest\MEDIA\temp\point3SSIM.txt", lumimask=1, scaled=0).Text("PointResize", y=66)
nne=SSIM(original, nnedi3, "\\myshare\Ingest\MEDIA\temp\nnedi3SSIM.csv", "\\myshare\Ingest\MEDIA\temp\nnedi3SSIM.txt", lumimask=1, scaled=0).Text("NNEDI3", y=66)
esr=SSIM(original, esrgan, "\\myshare\Ingest\MEDIA\temp\esrganSSIM.csv", "\\myshare\Ingest\MEDIA\temp\esrganSSIM.txt", lumimask=1, scaled=0).Text("ESRGAN", y=66)

#Preview
a=StackHorizontal(original, pnt)
b=StackHorizontal(nne, esr)

StackVertical(a,b)


And here are the images stacked as:

Original - PointResize
NNEDI3 - ESRGAN

I'm gonna pick one just to show why I'm sticking with NNEDI3:

https://i.imgur.com/A1b2eL6.png



Images collection:

Img1 (https://i.imgur.com/W7SXavj.png) - Img2 (https://i.imgur.com/yaRuaPC.png) - Img3 (https://i.imgur.com/NNmITRR.png) - Img4 (https://i.imgur.com/aUZs6iN.png) - Img5 (https://i.imgur.com/LEMrB9c.png) - Img6 (https://i.imgur.com/GCBFRkW.png) - Img7 (https://i.imgur.com/sBeZpS8.png) - Img8 (https://i.imgur.com/qtnf2hp.png) - Img9 (https://i.imgur.com/RgzDP6M.png) - Img10 (https://i.imgur.com/zxOJ1O7.png) - Img11 (https://i.imgur.com/Wcedbil.png) - Img12 (https://i.imgur.com/5rXpyBG.png) - Img13 (https://i.imgur.com/74H0zwA.png) - Img14 (https://i.imgur.com/ovQi3HV.png) - Img15 (https://i.imgur.com/KdWAAWo.png) - Img16 (https://i.imgur.com/NC8rIRR.png)

I guess I'm gonna stick with NNEDI3 for a while longer...

[Reel.Deel]

The result of that model looks very artificial. Have you tried any other models? I'd be worried about temporal consistency also. For real world video, the proprietary models from Topaz are good. Unfortunately those onnx models are housed in a password protected zip file :(.

[Emulgator]

Delivering a good choice from 186GB trained models Topaz (with Proteus v3 until 3.0.6 at least)
still had the "ugly face syndrom" when guessing at maybe-face-content from less than 20 pixel size.
For Proteus v4 this is promised to improve.
Some more training on face guessing from such small pixel patches should do.
But: Topaz VEAI 3 had Avisynth support killed, and users report many hassles.
My paid updates ended with 3.0.6, for the time being I wasn't willing to spend again, so I stick to last Topaz 2.6.4.

23.10.2023 23:21 Just today I came across my post by chance and oops:
Sorry for being completely OT. How did that get here ?
Must have been tired a bit on 26.03.2023 03:01, will try to move that where it belongs.

[anton_foy]

convertbits(32)
converttoplanarrgb()

mlrt_ncnn(network_path="C:\Program Files (x86)\AviSynth+\plugins64\ml_\models\1x-Film-Degrainer-1-000.onnx", fp16=true, builtin=false, tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8, list_gpu=true)


This loads the model fine and many (other models aswell) but it does nothing, it shows the original untouched clip. How to set the strength?
Or is it something else Im missing?

[kedautinh12]

You can delete tilesize_w=width/4, tilesize_h=height/4, overlap_w=8, overlap_h=8 if your GPU stronger than Reel.Deel's GPU

[Reel.Deel]

@anton_foy

In your script you have list_gpu=true, try setting it to false. As for a strength setting, there is none.

[anton_foy]

@anton_foy

In your script you have list_gpu=true, try setting it to false. As for a strength setting, there is none.

Thanks yes it works now! But really slow even with the RTX 4070.

[kedautinh12]

Did you try delete my recommended parameters too?

[anton_foy]

Did you try delete my recommended parameters too?

Thanks I will try this too. So by adjusting tilesize and overlap it can work for slower cards but slower processing rather than the default values for a faster card?

[kedautinh12]

Yeah, I tried delete it and the plugin will faster cause it will use more GPU memory. Try change fp16 to false too. If you meet error, you can change back to true

[Reel.Deel]

Thanks I will try this too. So by adjusting tilesize and overlap it can work for slower cards but slower processing rather than the default values for a faster card?

Using tilesize just means that the image is divided into sections, and because of overlap you end up processing more pixels. I don't think setting fp16 to true will have any negative effects on speed, even on higher end GPUs.

[anton_foy]

Thanks I tried both of your suggestions but it is impossibly slow and I tried Vapoursynth vs-scunet on the same clip and 30 frames would take over 1 hour. So I guess this processing UHD with these models is out of the question for now.

[kedautinh12]

Waiting for cuda ver
https://github.com/Asd-g/avs-mlrt/issues/7#issuecomment-1771187328

[anton_foy]

@kedautinh12
Thanks for the info good news.
Something I am wondering about is the num_streams param. Where to find the specification about this for my GPU? May this speed up if I put this to max for my card?

[Selur]

The number of stream depends on the resolution of your input and the VRAM that is used with model X and 1 stream.
If you use a model with one stream and only a fraction of your VRAM is used, you can run multiple streams in parallel to speed things up.
=> increasing the streams will cost VRAM, but can speed things up
So if either the input resolution or the used model changes, the VRAM usage and thus the amount of streams your card can handle can change.

[anton_foy]

The number of stream depends on the resolution of your input and the VRAM that is used with model X and 1 stream.
If you use a model with one stream and only a fraction of your VRAM is used, you can run multiple streams in parallel to speed things up.
=> increasing the streams will cost VRAM, but can speed things up
So if either the input resolution or the used model changes, the VRAM usage and thus the amount of streams your card can handle can change.

Thanks I will play around with this.

[kedautinh12]

Would like to see SwinIR models:
https://github.com/JingyunLiang/SwinIR/releases
https://openmodeldb.info/?q=SwinIR

[Dogway]

Yes, me too. I tried in vapoursynth and SwinIR didn't work for me there either. Looking for a solid photo upscaler.

[kedautinh12]

For Vapoursynth, why you don't ask HolyWu?
https://github.com/HolyWu/vs-swinir/issues?q=is%3Aopen+is%3Aissue

[Selur]

https://github.com/HolyWu/vs-swinir works fine here,...

[Dogway]

Ok, I thought the SwinIR models in pth format were supposed to work. Thanks both.

[kedautinh12]

Would like to see CodeFormer models :D
https://github.com/sczhou/CodeFormer/releases

[Selur]

Assuming some folks here use those models on normal content, how do you deal with temporal inconsistencies?
(like this: https://forum.videohelp.com/attachments/75185-1701473592/Face.mp4)
Personally, I use QTGMC with input type 3 to lessen this, but I was wondering what others do.

Cu Selur

[kedautinh12]

Would like to see UniScale models
https://openmodeldb.info/?q=Uniscale

[Selur]

You can convert those with chaiNNer (https://github.com/chaiNNer-org/chaiNNer),... (these are also included in the Hybrid onnx models collection)

[kedautinh12]

Would like to see SRFormer models
https://openmodeldb.info/?q=SRFormer

[Selur]

Would like to see UniScale models
Those seem to work fine here with vs-mlrt. (they are rather slow)
Would like to see SRFormer models
Best post such request over in the github issue tracker.

Cu Selur

[kedautinh12]

I can see it in other model. They use srformer to base for trained on
https://github.com/Phhofm/models

[kedautinh12]

Would like to see Cutie models
https://github.com/hkchengrex/Cutie?tab=readme-ov-file

[takla]

Actually it runs "ok" as long as you're not using any of the broken models from the dropbox link (I haven't tested them all, but 4 of 4 gave an error)
Should be fine if converted manually with "opset 17"

[takla]

Here are some models (https://www.mediafire.com/file/pocrytgzzejlsan/models.zip/file) which I've converted to onnx (Mostly for Cartoons/Anime)
I can vouche for all of them working (tested on AMD GPU)
Original sources can be found here (https://openmodeldb.info/)

2x_AniScale2S_Compact_i8_60K
Very good with BluRay sources

2x-LD-Anime-Compact
Amazing with high quality DVDs

2x_Garfieldjr_span48
Very Good for mediocre DVD quality

Example script:
LWLibavVideoSource("INPUT.mkv")
ConvertBits(32).ConvertToPlanarRGB()
mlrt_ort(network_path="C:\Program Files (x86)\AviSynth+\plugins64+\models2\2x-LD-Anime-Compact.onnx", builtin=false, provider="dml")