Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th April 2024, 09:35   #2921  |  Link
FTLOY
Friend of a friend..
 
FTLOY's Avatar
 
Join Date: Feb 2023
Posts: 290
Quote:
Originally Posted by Dogway View Post
Also try to use UHDHalf=true, mvtools2 might not handle 16-bit UHD resolution nicely.


Will be interested to hear what DTL might have say about statement, as he has been busy working on mvtools2..
FTLOY is offline   Reply With Quote
Old 29th April 2024, 12:30   #2922  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,161
Not nicely means performance is not great (also with blocksize of 8x8 even lower).

AVS cache size estimation for 4K 16bit (even in 4:4:4) and largest 'super' frame size with pel=2 (about 4x larger) - 8 Msamples of 6bytes x4 = 192 MB per frame. With 16threads and 10 frames in the cache per each thread - 25 GB in cache. Really big.

So it may be better to reduce the threads number for mvtools (AVS+ threads number of frames-based MT controlled by Prefetch(x,y) ) and if possible - number of frames in the AVS cache. And to keep more performance - enable internal MT via AVSTP.

I think the number of frames in the cache is expected to be controlled by Prefetch(M,N) where N is the number of frames. But I do not see if it really works.

In the sources it really has second param of frames -
https://github.com/pinterf/AviSynthP...cript.cpp#L264
{ "Prefetch", BUILTIN_FUNC_PREFIX, "c[threads]i[frames]i", Prefetcher::Create },


So you can try to test different Prefetch(threads, frames) settings too. With frames from 1. If it is not working - it is subject to report to AVS core developers too.

You can try to see memory usage with AVSmeter report at testing your total script. If different 'frames' Prefetch() settings the amount of RAM usage does not change - it looks like a bug to be fixed.

Last edited by DTL; 29th April 2024 at 12:34.
DTL is offline   Reply With Quote
Old 29th April 2024, 12:39   #2923  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,752
Quote:
Originally Posted by DTL View Post
I think the number of frames in the cache is expected to be controlled by Prefetch(M,N) where N is the number of frames. But I do not see if it really works.
It's been working since day one. The problem with current processors is that the default amount of prefetched frames is double the amount of threads, which then easily leads to excessive memory usage with UHD sources. The amount of prefetched frames is much less crucial than the amount of threads what comes to encoding performance, unless you work on near realtime encodes.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 29th April 2024, 13:39   #2924  |  Link
FTLOY
Friend of a friend..
 
FTLOY's Avatar
 
Join Date: Feb 2023
Posts: 290
@ DTL & Boulder,

Could either of you show me what you would change in script, please.

I'm running with Prefetch(12), but it doesn't need to be in the script, with RipBot.

Running 7950X, 5950X etc.

Or is it pointless ?
FTLOY is offline   Reply With Quote
Old 29th April 2024, 14:33   #2925  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,161
"I'm running with Prefetch(12)"

Try to start with Prefetch(12,1) . And with RipBot you need to find where it sends Prefetch(x,(y?)) line to AVS (if it really sends) and also edit its second param to 1 or something low as 1,2,3,etc. If second param not provided it may be up to double of threads number ? So your Prefetch(12) may be internally Prefetch(12,24) and use too much RAM for cache.

Or try to ask RipBot developers how to set prefetch frames number if it sends Prefetch() line to AVS after user's script.

As some hack you can try to add

return video.Prefetch(12,1)

as last line to the script. But it may broke RipBot if it uses special name of clip to return (video ?) or other.

Also as some experiment you can try to add .Prefetch(12,1) to each filter call (except source) so it may force AVS to set these prefetch params at least for these filters. Like
Code:
video=ConvertBits(video,16).Prefetch(12,1)
pre=DGHDRtoSDR(video,mode="pq",white=2000,gamma=1/2.4,tm=1.0).ex_Median(mode="IQMST",thres=255).Prefetch(12,1)
video=STTWM(video,sw=50,tw=50,aw=100,sthres=5,tthres=5).Prefetch(12,1)
video=SMDegrain(video,tr=2,thSAD=200,thSADC=100,UHDHalf=false,limit=255,contrasharp=true,prefilter=pre,str=5.0,refinemotion=true).Prefetch(12,1)

Last edited by DTL; 29th April 2024 at 17:17.
DTL is offline   Reply With Quote
Old 30th April 2024, 05:26   #2926  |  Link
FTLOY
Friend of a friend..
 
FTLOY's Avatar
 
Join Date: Feb 2023
Posts: 290
Quote:
Originally Posted by DTL View Post
"I'm running with Prefetch(12)"

Try to start with Prefetch(12,1) . And with RipBot you need to find where it sends Prefetch(x,(y?)) line to AVS (if it really sends) and also edit its second param to 1 or something low as 1,2,3,etc. If second param not provided it may be up to double of threads number ? So your Prefetch(12) may be internally Prefetch(12,24) and use too much RAM for cache.

Or try to ask RipBot developers how to set prefetch frames number if it sends Prefetch() line to AVS after user's script.

As some hack you can try to add

return video.Prefetch(12,1)



as last line to the script. But it may broke RipBot if it uses special name of clip to return (video ?) or other.

Also as some experiment you can try to add .Prefetch(12,1) to each filter call (except source) so it may force AVS to set these prefetch params at least for these filters. Like
Code:
video=ConvertBits(video,16).Prefetch(12,1)
pre=DGHDRtoSDR(video,mode="pq",white=2000,gamma=1/2.4,tm=1.0).ex_Median(mode="IQMST",thres=255).Prefetch(12,1)
video=STTWM(video,sw=50,tw=50,aw=100,sthres=5,tthres=5).Prefetch(12,1)
video=SMDegrain(video,tr=2,thSAD=200,thSADC=100,UHDHalf=false,limit=255,contrasharp=true,prefilter=pre,str=5.0,refinemotion=true).Prefetch(12,1)
Sadly, none of this made any difference to the RAM usage, even the RipBot dev suggested something, but again, that made no difference.

So it's either not using this script, OR buying more RAM.

One thing I should mention is, as RipBot can use Distributed Encoding, you can "load up" the CPU until it basically slows the whole process down, and I generally process 2 chunks at a time, which maxes out the RAM, IF I only process 1 chunk at a time, the RAM usage is more than acceptable.

I can simply run more encoding servers to "pick up the slack"....


So anyone else using this script would not have a RAM usage issue, except maybe with StaxRip, which offers a form of chunk encoding, but only on the same PC.


Last edited by FTLOY; 30th April 2024 at 05:41.
FTLOY is offline   Reply With Quote
Old 30th April 2024, 13:07   #2927  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,161
" none of this made any difference to the RAM usage"

It is really a sad sign of possible more bugs in the prefetcher exists. I will try to test at my use cases and if it also not works as expected - it is a subject to new issue at AVS+ development.
DTL is offline   Reply With Quote
Old 30th April 2024, 13:22   #2928  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,752
I would first test the script (and post it here) with different Prefetch parameters in AVSMeter64 before spending too much time faffing about.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...

Last edited by Boulder; 30th April 2024 at 13:27.
Boulder is offline   Reply With Quote
Old 30th April 2024, 17:29   #2929  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,161
It looks like subject of long testing:

With simple internal script
Code:
ColorBars()

TurnRight()
#Prefetch(2,1)
Prefetch(2)
RAM usage difference is big between Prefetch[2] and Prefetch[2,1] host CPU is 2 cores E7500. About 20 vs 13 MB. But with my typical test script with lots of mvtools filters calls it close to several % with frame size 800x480 - about 640 and 630 MB. So Prefetct[2,1] is really working to make RAM usage a bit lower - but not as low as expected. Need to test with UHD frame sizes and pel=2 later at host with may RAM and many cores CPU. Also it is better to test with real SMDegrain script running. I typically can not install all required plugins to test it. So if someone have all environment configured to run SMDegrain it is faster to test at that host.
DTL is offline   Reply With Quote
Old 30th April 2024, 17:45   #2930  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,752
I tested a UHD source with my denoiser script with some MVTools stuff, outputting 16-bit data in and out. 32 threads in use.
If I put my resizer there, the usage would be much higher I think.

frames 1: 4560 MB
frames 4: 6196 MB
frames 8: 8580 MB
frames 16: 13097 MB
frames 32: 19188 MB
default, frames 64: 22014 MB (hitting some limit? Max is set to 30000 MB)
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 6th May 2024, 06:15   #2931  |  Link
LouieChuckyMerry
Registered User
 
LouieChuckyMerry's Avatar
 
Join Date: Feb 2014
Posts: 370
Hi Dogway, I hope you're well (and am glad you're back) . After transitioning from your original SMDegrain Mod to real.finder's continuation, I finally have your latest SMDegrain Mod (v4.6.0d) working but can't figure out the equivalent of "LSB_In=True" and "LSB_Out=True" ("n16=True" and "n16_Out=True" with real.finder's versions). My typical SMDegrain, well, NotSMDegrain, call is

Code:
SMDegrain(TR=1~5,ThSAD=100~900,RefineMotion=True,Plane=0,Chroma=False,n16=True,n16_Out=True)
Thanks for any help.
LouieChuckyMerry is offline   Reply With Quote
Old 6th May 2024, 10:59   #2932  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,161
Some hacks around 16bit were required for old versions of mvtools. With 2.7.45 (46) you can try simply feed 10..16 bit input without any additional params and expect it is processed unchanged and output the same.
DTL is offline   Reply With Quote
Old 6th May 2024, 17:14   #2933  |  Link
LouieChuckyMerry
Registered User
 
LouieChuckyMerry's Avatar
 
Join Date: Feb 2014
Posts: 370
DTL: Thanks for your reply. I'm just a hobbyist, so my understanding of AviSynth is quite limited and my use of it's very simple. Typically, I'll take either an 8-bit or 10-bit source and process it with the script

Code:
SetFilterMTMode("Default_MT_Mode",2)
SMDegrain(TR=1~5,ThSAD=200~900,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)
PreFetch(X,depending on which computer)
outputting to 10bit x264. Sometimes I'll use Spotless() as a prefilter, but that's about as complex as I get. So, to achieve processing the above in 16-bit would I add "ConvertBits(Bits=16)" to the script here

Code:
SetFilterMTMode("Default_MT_Mode",2)
ConvertBits(Bits=16)
SMDegrain(TR=1~5,ThSAD=200~900,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)
PreFetch(X,depending on which computer)
Thanks for your help.
LouieChuckyMerry is offline   Reply With Quote
Old 7th May 2024, 06:13   #2934  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,752
Quote:
Originally Posted by LouieChuckyMerry View Post
So, to achieve processing the above in 16-bit would I add "ConvertBits(Bits=16)" to the script here

Code:
SetFilterMTMode("Default_MT_Mode",2)
ConvertBits(Bits=16)
SMDegrain(TR=1~5,ThSAD=200~900,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)
PreFetch(X,depending on which computer)
That would most likely be enough. You can doublecheck the output video format by adding Info() after the SMDegrain line and opening the script in VDub for example.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 7th May 2024, 09:08   #2935  |  Link
Lucky38
Registered User
 
Join Date: Mar 2024
Posts: 10
Quote:
Originally Posted by LouieChuckyMerry View Post
Hi Dogway, I hope you're well (and am glad you're back) . After transitioning from your original SMDegrain Mod to real.finder's continuation, I finally have your latest SMDegrain Mod (v4.6.0d) working but can't figure out the equivalent of "LSB_In=True" and "LSB_Out=True" ("n16=True" and "n16_Out=True" with real.finder's versions). My typical SMDegrain, well, NotSMDegrain, call is

Code:
SMDegrain(TR=1~5,ThSAD=100~900,RefineMotion=True,Plane=0,Chroma=False,n16=True,n16_Out=True)
Thanks for any help.

Hi,

could someone explain how SMDegrain will known when use TR=1 or TR=5 and the same for ThSAD?

i did not see in documentation any information that these parameters can have range value....
Lucky38 is offline   Reply With Quote
Old 7th May 2024, 09:34   #2936  |  Link
FTLOY
Friend of a friend..
 
FTLOY's Avatar
 
Join Date: Feb 2023
Posts: 290
Quote:
Originally Posted by Lucky38 View Post
Hi,

could someone explain how SMDegrain will known when use TR=1 or TR=5 and the same for ThSAD?

i did not see in documentation any information that these parameters can have range value....
Here's an example...

https://forum.doom9.org/showthread.p...92#post2001092 post #2917
FTLOY is offline   Reply With Quote
Old 7th May 2024, 15:01   #2937  |  Link
StvG
Registered User
 
Join Date: Jul 2018
Posts: 465
Quote:
Originally Posted by Boulder View Post
Quote:
Originally Posted by LouieChuckyMerry
So, to achieve processing the above in 16-bit would I add "ConvertBits(Bits=16)" to the script here

Code:
SetFilterMTMode("Default_MT_Mode",2)
ConvertBits(Bits=16)
SMDegrain(TR=1~5,ThSAD=200~900,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)
PreFetch(X,depending on which computer)
That would most likely be enough. You can doublecheck the output video format by adding Info() after the SMDegrain line and opening the script in VDub for example.
F3KDB doesn't support native 16-bit (only stacked, interleaved). Repalce it with neo_f3kdb.
StvG is offline   Reply With Quote
Old 8th May 2024, 06:36   #2938  |  Link
Lucky38
Registered User
 
Join Date: Mar 2024
Posts: 10
Quote:
Originally Posted by FTLOY View Post

hi FTLOY,

thank you for your replay but this is not the explanation i was hope to get...

I use SMDegrain but i havn't seen that TR=1~5 and ThSAD=200~900 can work with range value...

That is why i was asked how does SMDegrain will known when use TR=1 or TR=2 ... or TR=5 ?
Lucky38 is offline   Reply With Quote
Old 8th May 2024, 07:22   #2939  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,752
Quote:
Originally Posted by Lucky38 View Post
hi FTLOY,

thank you for your replay but this is not the explanation i was hope to get...

I use SMDegrain but i havn't seen that TR=1~5 and ThSAD=200~900 can work with range value...

That is why i was asked how does SMDegrain will known when use TR=1 or TR=2 ... or TR=5 ?
That was just to mark an example of range. You need to set a fixed TR and ThSAD to your liking.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 8th May 2024, 18:43   #2940  |  Link
Lucky38
Registered User
 
Join Date: Mar 2024
Posts: 10
Quote:
Originally Posted by Boulder View Post
That was just to mark an example of range. You need to set a fixed TR and ThSAD to your liking.
thanks for clarification.
Lucky38 is offline   Reply With Quote
Reply

Tags
avisynth, dogway, filters, hbd, packs

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.