Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th February 2023, 18:07   #21  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
Can ask Asd-g about RIFE??

Last edited by kedautinh12; 26th February 2023 at 18:20.
kedautinh12 is offline   Reply With Quote
Old 26th February 2023, 18:09   #22  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Quote:
Originally Posted by poisondeathray View Post
Many of them are difficult to get working (at least for me, on Windows) . An interesting one that I actually got to work, and is video inpainting (temporal +/- 3 frames) is OPN . Worked ok for some sources, not for others. YMMV
https://github.com/seoungwugoh/opn-demo
Thanks a lot, yes I wonder why temporal information isn't used more frequently in inpainting. I mean it has huge potential not only in restoration but also VFX. I will give it a stab when I'm in Win10.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 26th February 2023, 18:27   #23  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by DTL View Post
And what is the current RIFE is based at ? Or it is separated engine for doing motion interpolation ?
I don't understand the question - RIFE is flow based frame prediction (intermediate frame(s) synthesized between 2 source frames) .

The video SR/deblurring/denoising algos mentioned are multiframe bidirectional ...n-3, n-2, n-1, n, n+1, n+2, n+3... you can set 30, 50 or more frames if you wanted to


Quote:
Some wiki https://www.svp-team.com/wiki/RIFE_AI_interpolation says it is based on IFNet neural network.
Also it shows TensorRT at NVIDIA cards can run RIFE about 2x faster in compare with ncnn/Vulkan implementation. So RIFE (IFNet ?) can be compiled (?) to different hardware APIs ?
Yes, different implementations are possible

Yes, TensorRT is usually the fastest implementation for all machine learning algorithms, not just rife .

Quote:

As for only storing frames in memory - the 1920x1080 frame in 12bytes per sample RGB 32bit float is about 24 MB only. So 1/4 of 1GB onboard RAM can keep 10 frames and it is very good already to study how motion is going between several frames.
GPU memory, not system memory . And it's not just the additive frame memory; pytorch /TensorRT/ NCNN, all of them - consume extra reserve memory for operations .


Quote:
I post an issue on some RIFE development project at github about possible changing RIFE algorithm for better denoise process - https://github.com/megvii-research/E...IFE/issues/304 but still no one reply. May be it require to search someone with understanding how RIFE work so the required changes can be implemented.
I haven't had a chance to look at it yet, but I'm not sure that RIFE would add much over for the mocomp over say mvtools2, and it would be slower
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 18:34   #24  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Object tracking is part of it. Also propogation and alignment.

There are typically catergorized into single image vs. video . "Video" algorithms takes >1 frame into account. All the ESRGAN / Real-ESRGAN derivatives are single image only.

Examples of "video" algorithms that use "propogation", "alignment" at runtime are VRT, RVRT, BasicVSR++, RealBasicVSR. At runtime is important because feature are matched in adjacent frames in the specific input video.

Beware video algorithms take lots of memory. Some of them have "tiling" options built in, but the results can be worse especially for operations like deblurring. The whole frame in +/- n frames should be compared on some tasks, otherwise you get discontinunities at the tile boundaries

I posted some examples of BasicVSR/BasicVSR++ in a few threads here and other forums. The provided model works on "textbook" clean cases. A big problem for is degradation step wasn't included , so it's not as generalizable to real world, compression cases. RealBasicVSR adds degradation, but the results are quite blurry / detail loss
Thanks for the thorough reply. I tried DPIR but had to downscale from uhd to 1280x720 for it to run and then it runs pretty fast but runs not at all above that resolution.

Even if it is only "spatial" denoising it can be used before MSuper()
Code:
spatialfilter.MSuper()
and then Mdegrain to "smooth" out the motion jitter.
Usually I do this with fft3dfilter and mdegrain but maybe there is some better way to stabilize the motion jitter/wobble caused by noise.

Very curious to try the noise models in the model_database link REEL.DEEL posted, especially the "1x_ISO_denoise_v1" and v2 and "Film-Degrainer_1-000".
anton_foy is offline   Reply With Quote
Old 26th February 2023, 18:37   #25  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by poisondeathray View Post
Ok, some very minor differences (only detectable on amplified differences) between avs, chainner using onnx and vs. pytorch in some cases, when settings are the same (e.g. tiles) . I think some of them might be due to the way it's converted to float or perhaps [0,1] clipping. The differences are so minor they are nothing to worry about
Thanks for testing, good to know that there are no big differences. That's the consensus I got from other places. When converting some models I get a warning "Backend: WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function." but from what I read, they are harmless.

Quote:
Originally Posted by poisondeathray View Post
Could be GPU memory ?

Did you try the tiling options ?
Probably, I need to experiment with those options.

Quote:
Originally Posted by Dogway View Post
Does anyone know of a good inpaint model? I only know 1x_NMKD-YandereInpaint_375000_G and 1x_sudo_inpaint_PartialConv2D_424000_G but not sure which one might be better for opaque logos on live action content.

1x_JPEG_60-80.pth can also be good for deblocking old codec sources, need to compare with 1x-Kim2091_DeJpeg.onnx.
I'll add those to the package.

Quote:
Originally Posted by DonCanjas View Post
Would like to see
  • 4x_eula_digimanga_bw_v2_nc1_307k
  • MangaJPEG
  • 1x_GainRESV3 (Aggro,Natural,Passive)
  • 1x_DitherDeleter-Smooth_104000_G
Will do.
Reel.Deel is offline   Reply With Quote
Old 26th February 2023, 18:37   #26  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Quote:
Originally Posted by kedautinh12 View Post
Can ask Asd-g about RIFE??
May be only RIFE developers https://arxiv.org/pdf/2011.06294v12.pdf understand how it is designed and how to change in into motion compensating engine based on reference frame. Not to interpolate between 2 given images.

Asd-g only make interface between RIFE engine and AVS+.
DTL is offline   Reply With Quote
Old 26th February 2023, 18:41   #27  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by anton_foy View Post
Thanks for the thorough reply. I tried DPIR but had to downscale from uhd to 1280x720 for it to run and then it runs pretty fast but runs not at all above that resolution.
You can use "tiling" .

Look at the avsi for tilesize and overlap options. For spatial denoising like DPIR, if you use large enough overlap, the differences are usually undetectable to eye (except with amplified differences) between single tile (full image), vs. say 2x2 grid - many tests were done with the vapoursynth version in the past - I'm assuming the avs version is the same


Quote:
Even if it is only "spatial" denoising it can be used before MSuper()
Code:
spatialfilter.MSuper()
and then Mdegrain to "smooth" out the motion jitter.
Usually I do this with fft3dfilter and mdegrain but maybe there is some better way to stabilize the motion jitter/wobble caused by noise.
Not just for denoising; all single image operations are prone temporal inconsistencies. Another option is QTGMC in progressive mode but it has downsides too, even if you adjust the settings

It would be nice to have other options for temporal smoothing

Last edited by poisondeathray; 26th February 2023 at 18:44.
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 18:58   #28  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Quote:
Originally Posted by poisondeathray View Post
The video SR/deblurring/denoising algos mentioned are multiframe bidirectional ...n-3, n-2, n-1, n, n+1, n+2, n+3... you can set 30, 50 or more frames if you wanted to

I haven't had a chance to look at it yet, but I'm not sure that RIFE would add much over for the mocomp over say mvtools2, and it would be slower
Do these algos mentioned have frame interpolation implementations (like framerate doubling, same as RIFE) so can also be tested for denoise performance/quality ?

The major issue of current motion compensation in mvtools is being limited to compensation of translate transform only. Also its motion estimation is not ideal because of noise (and noise can not be removed before motion estimation - it is 'egg and chicken' problem same as mentioned in that RIFE document).

So from the neural networks denoise (motion compensation for 'static' blending engines) expected:
1. Compensate for more real transforms (scale/rotate/skew/...)
2. To be less sensitive to natural noise in the sources to analyse (using variable size / variable shape areas to analyse in a big enough sequence of frames or better in total scene or even movie).

Current mvtools approach to decrease block size to have better and finer motion tracking also degrades its ability to correctly track noise-damaged blocks. So when signal-to-noise ratio come close to 1 or fall below 1 typical block-based motion search engines (also limited to 2 frames only) give very large errors and mostly useless.

So if in current quickly dying from degradation civilization we have very limited and dying programmers resources to design better motion search engines we try to look for 'hardware/machine learning' tools that can replace current poor programmers resources by 'brute force' of GBytes memory and TeraFlops+ accelerators and 'neural networks with ability to learn'. The performance is not very critical if it can provide significantly better quality in compare with current block-based simple motion search in mvtools. Some titles wait years and decades before being processed in yet another rip for release in possibly more or less better quality. So if it even will be processed several days/weeks/months it is not real physical problem.

Last edited by DTL; 26th February 2023 at 19:04.
DTL is offline   Reply With Quote
Old 26th February 2023, 19:07   #29  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by DTL View Post
Do these algos mentioned have frame interpolation implementations (like framerate doubling, same as RIFE) so can also be tested for denoise performance/quality ?
VRT does; it's very very slow , and consumes large amounts of memory . It has 3 tiling , 3 overlap parameters, frame(temporal), height, width . (note that it's height first, not width)

https://github.com/JingyunLiang/VRT


Quote:
The major issue of current motion compensation in mvtools is being limited to compensation of translate transform only. Also its motion estimation is not ideal because of noise (and noise can not be removed before motion estimation - it is 'egg and chicken' problem same as mentioned in that RIFE document).
Prefilter can sometimes help, but yes chicken vs. egg.
poisondeathray is offline   Reply With Quote
Old 26th February 2023, 19:34   #30  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Quote:
Originally Posted by poisondeathray View Post
Prefilter can sometimes help, but yes chicken vs. egg.
Prefilter may degrade some valueable details for output. Also prefilter may increase motion search error. So I not like to use 'simple' prefilter before motion-compensated averaging.

"https://github.com/JingyunLiang/VRT"

It looks VRT still not have interface to Avisynth ? And can not be loaded via AviSynth+'s avs-mlrt plugin ?

Last edited by DTL; 26th February 2023 at 19:41.
DTL is offline   Reply With Quote
Old 26th February 2023, 19:48   #31  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by DTL View Post

It looks VRT still not have interface to Avisynth ? And can not be loaded via AviSynth+'s avs-mlrt plugin ?
No avs interface, and cannot be loaded with avs-mlrt
poisondeathray is offline   Reply With Quote
Old 27th February 2023, 00:05   #32  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Uploaded a handful of models that were requested here and also a few other's of Zarxrax's models (2x_LD-Anime_Compact_330k_net_g, 1x_Dotzilla_SuperUltraCompact_300k_net_g, 1x_Dotzilla_UltraCompact_182k_net_g).

@dogway, 1x_NMKD-YandereInpaint_375000_G is not available, do you have DL link?

Edit

Quote:
Originally Posted by poisondeathray View Post
Thanks!

Last edited by Reel.Deel; 27th February 2023 at 00:16.
Reel.Deel is offline   Reply With Quote
Old 27th February 2023, 00:10   #33  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by Reel.Deel View Post
1x_NMKD-YandereInpaint_375000_G is not available, do you have DL link?
https://www.mediafire.com/file/fp8gd...npaint.7z/file
poisondeathray is offline   Reply With Quote
Old 27th February 2023, 02:05   #34  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
Quote:
Originally Posted by poisondeathray View Post
I mention it because sometimes an earlier checkpoint produces better results. Here are all the checkpoints for DeGif that I have. Credits to NMKD (n00mkrad)
https://www.mediafire.com/file/2jzoq...+DeGIF.7z/file

Just curious :
1) how long does it take to convert ?
2) are results bit identical to the pytorch version?
can you convert it to onnx??
kedautinh12 is offline   Reply With Quote
Old 27th February 2023, 02:15   #35  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,345
Quote:
Originally Posted by kedautinh12 View Post
can you convert it to onnx??
https://www.mediafire.com/file/51mae...G_onnx.7z/file
poisondeathray is offline   Reply With Quote
Old 27th February 2023, 02:32   #36  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
Thanks
kedautinh12 is offline   Reply With Quote
Old 27th February 2023, 02:52   #37  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by poisondeathray View Post
Could be GPU memory ?

Did you try the tiling options ?
I was able to get it running using those options, although at a snails pace. I guess my GPU is not adequate for the bigger models (1050 now, I plan on upgrading to a 2070 in the upcoming months).
Reel.Deel is offline   Reply With Quote
Old 27th February 2023, 05:22   #38  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
All avs+ plugins collection by me, sort by architecture like Asd-g. @Reel.Deel i think you can sort like that for easy findout
https://drive.google.com/file/d/11Vk...w?usp=drivesdk
kedautinh12 is offline   Reply With Quote
Old 27th February 2023, 09:16   #39  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by poisondeathray View Post
You can use "tiling" .

Look at the avsi for tilesize and overlap options. For spatial denoising like DPIR, if you use large enough overlap, the differences are usually undetectable to eye (except with amplified differences) between single tile (full image), vs. say 2x2 grid - many tests were done with the vapoursynth version in the past - I'm assuming the avs version is the same




Not just for denoising; all single image operations are prone temporal inconsistencies. Another option is QTGMC in progressive mode but it has downsides too, even if you adjust the settings

It would be nice to have other options for temporal smoothing
Thanks! Yes tiling and overlap, I need to get back to my comp and try this, I had good results too with QTGMC's denoising in the past but it left some noise yet calmed down. Could you or someone convert 1x_ISO_denoise_v2? I would like to try all of the denoisers/degrainers in there but if that is too much I can try to convert them myself. Any link to the conversion software or does it cost?

Edit: thank you so much Reel.Deel great to see these kind of filters coming out for avs+!

Last edited by anton_foy; 27th February 2023 at 13:37.
anton_foy is offline   Reply With Quote
Old 27th February 2023, 12:26   #40  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,352
Thanks for the conversions Reel.Deel!
Maybe HiNet can be converted? It had really astonishing results for scratches.
And just asking out of the blue, not sure if a film grain model as such exists, that is to add filmic grain, it can be an alternative to chickendream which is too slow. EDIT: yes, found 1x_Filmify4K_v2_325000_G but grain is not of great quality...
I also had 4x_Valar_v1 which is good for photo upscales it seems, but as with many of these it might require perfect artifact free downscaled content.

Also as a suggestion it might be interesting to try 2x_Waifaux-NL3-SuperLite_latest_G, as it might be a faster alternative to current solutions like w2xncnnvk.

EDIT: Tested 1x_JPEG_60-80 and 1x-Kim2091_DeJpeg and got a system freeze, black screen and AvsPmod crash. Well anyway I compared them to Oyster and they paled against it. Since Oyster is so slow, it would be interesting to train a model based on it with my script from this post and probably a deblur model on it. Maybe if I have time in the future I can do such thing.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 27th February 2023 at 14:16.
Dogway is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:12.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.