Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum Avisynth+
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 21st March 2015, 19:19 #1021  |  Link Boulder Pig on the wing     Join Date: Mar 2002 Location: Finland Posts: 5,187 Hey, I'm in luck today I found out that the very first thing done by my cleaner function is the one that starts acting up. Code: ```SetFilterMTMode("DEFAULT_MT_MODE", 2) DGSource("hotfuzztest.dgi") Flux5framesT(th=2,chromamotion=true) Prefetch(4)``` and Flux5framesT along with what it needs: Code: ```function Flux5framesT(clip c, int "th", int "thC", bool "chromamotion") { th = default(th,7) thC = default(thC, (chromamotion == true) ? th : 0) med = c.TMedian2(chromamotion=chromamotion) avg = c.temporalsoften(2,th,thC,24,2) medD = mt_makediff(c,med,U=chromamotion?3:1,V=chromamotion?3:1) avgD = mt_makediff(c,avg,U=chromamotion?3:1,V=chromamotion?3:1) DD = mt_lutxy(medD,avgD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=chromamotion?3:1,V=chromamotion?3:1) output = c.mt_makediff(DD,U=chromamotion?3:1,V=chromamotion?3:1) return output } function TMedian2(clip c, bool "chromamotion") { Median2( c.selectevery(1,-2), c.selectevery(1,-1), c, c.selectevery(1,1), c.selectevery(1,2), chromamotion=chromamotion ) } Function Median2(clip "input_1", clip "input_2", clip "input_3", clip "input_4", clip "input_5", string "chroma", bool "chromamotion") {# median of 5 clips from Helpers.avs by G-force chromamotion = default(chromamotion, true) chroma = default(chroma, (chromamotion == true) ? "process" : "copy first") #default is "process". Alternates: "copy first" or "copy second" #MEDIAN(i1,i3,i5) Interleave(input_1,input_3,input_5) chroma == "process" ? Clense() : Clense(grey=true) #chroma == "process" ? MedianBlurT(0,0,0,1).merge(last,0.5).MedianBlurT(0,0,0,1) : MedianBlurT(0,-1,-1,1).merge(last,0.5).MedianBlurT(0,-1,-1,1) m1 = selectevery(3,1) #MAX(MIN(i1,i3,i5),i2) m2 = input_1.MT_Logic(input_3,"min",chroma=chroma).MT_Logic(input_5,"min",chroma=chroma).MT_Logic(input_2,"max",chroma=chroma) #MIN(MAX(i1,i3,i5),i4) m3 = input_1.MT_Logic(input_3,"max",chroma=chroma).MT_Logic(input_5,"max",chroma=chroma).MT_Logic(input_4,"min",chroma=chroma) Interleave(m1,m2,m3) chroma == "process" ? Clense() : Clense(grey=true) #chroma == "process" ? MedianBlurT(0,0,0,1).merge(last,0.5).MedianBlurT(0,0,0,1) : MedianBlurT(0,-1,-1,1).merge(last,0.5).MedianBlurT(0,-1,-1,1) selectevery(3,1) chroma == "copy first" ? last.MergeChroma(input_1) : chroma == "copy second" ? last.MergeChroma(input_2) : last Return(last) }``` I think these two plugins are the only ones that are needed: https://drive.google.com/file/d/0Bze...ew?usp=sharing EDIT: when replacing the two "chroma ==" lines with the MedianBlurT version in the Median2 function, I just get a crash when opening the script. EDIT2: In fact, in AVS+ the function doesn't seem to do anything. Comparing it with the original clip doesn't show any differences, with SEt's MT build the changes are obvious. __________________ And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... Last edited by Boulder; 21st March 2015 at 19:34.
21st March 2015, 20:54   #1022  |  Link
ultim
AVS+ Dev

Join Date: Aug 2013
Posts: 359
Quote:
 Originally Posted by Boulder I was thinking more on the lines of finding out if there are possible problems in memory handling as ultim said he's looking to make the cache handling more robust. Then again, I seem to get the script to stall even with SetMemoryMax(256) and two threads.
Yes the caches need some fine tuning. Currently they work great until memory starts running out.

It would certainly help a lot to be able to test with your scripts. If you use a lot of helper scripts, just post them too If you don't want to make them public, maybe you can PM or e-mail them to me?

EDIT: Oh, you made a new post while I wrote my own. Thanks, I'll check it out.
__________________
AviSynth+

Last edited by ultim; 21st March 2015 at 20:57.

22nd March 2015, 06:50   #1023  |  Link
colours
Registered User

Join Date: Mar 2014
Posts: 308
Quote:
 Originally Posted by Boulder I found out that the very first thing done by my cleaner function is the one that starts acting up. Code: `reallylongscript`
I know this is not very relevant, but I'm seeing a whole lot of optimisation potential there. Not tested, but should produce identical results and be faster.

Code:
```function Flux5framesT(clip c,int "th",int "thC",bool "chromamotion")
{
th  = default(th, 7)
thC = default(thC, chromamotion ? th : 0)
med = chromamotion ? ytouv(c.utoy8().median5t(), c.vtoy8().median5t(), c.median5t()) : c.median5t().mergechroma(c)
avg = c.temporalsoften(2, th, thC, 24, 2)
output = interleave(c, med, avg).clense(grey=!chromamotion).selectevery(3,1)
return output
}

function median5t(clip src)
{ # from here
function min(clip a, clip b) {return mt_logic(a, b, mode="min")}
function max(clip a, clip b) {return mt_logic(a, b, mode="max")}
src
last + trim(framecount()-1,-1).loop(5)
bcmin = min(SelectEvery(2, -1), SelectEvery(2, 0))
bcmax = max(SelectEvery(2, -1), SelectEvery(2, 0))
demin = bcmin.SelectEvery(1, 1)
demax = bcmax.SelectEvery(1, 1)
x = max(bcmin, demin)
y = min(bcmax, demax)
a = SelectEvery(2, -2)
f = SelectEvery(2, 3)
Interleave(a, x, y, f).Clense(grey=true).SelectEvery(4, 1, 2)
trim(0,length=src.framecount())
}```
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

 22nd March 2015, 10:06 #1024  |  Link Boulder Pig on the wing     Join Date: Mar 2002 Location: Finland Posts: 5,187 Thanks for those, I'll test them and replace the old ones if they are indeed faster The old ones have sometimes been posted here on this forum and I've borrowed them as a small pre-cleaner function to the MC'd denoiser. __________________ And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon...
 22nd March 2015, 11:47 #1025  |  Link Boulder Pig on the wing     Join Date: Mar 2002 Location: Finland Posts: 5,187 Flux5FramesT needed just the line "chromamotion = default(chromamotion, true)" so I added that one since I very rarely disable chroma motion detection. However, the initial problem still remains, a test with AVSMeter stalls the process at some point. I'm also getting CPU usage below 25% (1 core) even with four threads, maybe that's something to look at (caching again?) __________________ And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon...
22nd March 2015, 11:47   #1026  |  Link
jones1913
random user

Join Date: May 2014
Location: #Neuland
Posts: 108
Quote:
 Originally Posted by noee I've been testing with AVS+/MT mode w/QTGMC. So far, I haven't checked numbers against SETs 2.6RC1, but fwiw, in 64bit version, AVS+ will complete an encode that fails with the older (JoshyD?) 64-bit build. Have you tried yet? If so, what kind of numbers do you see?
Do you mean that you run QTGMC with 64 bit AVS? Until now I assumed that not all needed plugins are available as 64 bit version.

Here are my results with QTGMC in MT mode on AMD FX 8320 (8 core):

AVS 2.6:
Code:
```SetMTMode(3, 4)
LWLibavVideoSource("sample.m2v")
SetMTMode(2)
QTGMC(Preset="Medium")
SelectEven()
Distributor()

AVSMeter 1.9.8.0 (x86)
AviSynth 2.60, build:Feb 20 2015 [03:16:45] (2.6.0.5)
Active MT Mode: 2

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      5.680 | 91306 | 43.94
Memory usage (phys | virt):     585 | 604 MB
CPU usage (average):            74%

Time (elapsed):                 00:00:39.919```
AVS+:
Code:
```SetFilterMTMode("DEFAULT_MT_MODE",2)
LWLibavVideoSource("sample.m2v")
QTGMC(Preset="Medium")
SelectEven()
Prefetch(4)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.676 | 114785 | 33.88
Memory usage (phys | virt):     398 | 414 MB
CPU usage (average):            56%

Time (elapsed):                 00:00:51.768```
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool

22nd March 2015, 12:43   #1027  |  Link
Stereodude
Registered User

Join Date: Dec 2002
Location: Region 0
Posts: 1,396
Quote:
 Originally Posted by jones1913 Do you mean that you run QTGMC with 64 bit AVS? Until now I assumed that not all needed plugins are available as 64 bit version.
I've got it working several years ago with AVIsynth x64. I don't recall it being much faster in 64-bit though. It was a giant pain to find x64 versions of all the plugins.

22nd March 2015, 13:03   #1028  |  Link
noee
Registered User

Join Date: Jan 2007
Posts: 530
Quote:
 Originally Posted by Stereodude I've got it working several years ago with AVIsynth x64. I don't recall it being much faster in 64-bit though. It was a giant pain to find x64 versions of all the plugins.
Yes, that seems to be the case. My scripts with QTGMC are quite simple and when it would work in 64bit, it was about 3% faster than 32bit MT (SET).
__________________
Win7Ult || RX560/4G || Ryzen 5

22nd March 2015, 13:05   #1029  |  Link
Groucho2004

Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
 Originally Posted by jones1913 Code: `Distributor()`
Don't put that in the script unless you specified "InvokeDistributor=0" in AVSMeter.ini.
That goes for pretty much every application that uses AVISynth(MT) except ffdshow and some piping tools that don't call Distributor() internally.

22nd March 2015, 13:46   #1030  |  Link
jones1913
random user

Join Date: May 2014
Location: #Neuland
Posts: 108
Quote:
 Don't put that in the script unless you specified "InvokeDistributor=0" in AVSMeter.ini.
I have indeed set "InvokeDistributor=0" because initially I had some problems get this to run with AVS+, so I tried several things.

Quote:
 That goes for pretty much every application that uses AVISynth(MT) except ffdshow and some piping tools that don't call Distributor() internally.
OK I didn't know that, thanks.
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool

22nd March 2015, 14:17   #1031  |  Link
chainik_svp
Registered User

Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
 Originally Posted by jones1913 Here are my results with QTGMC in MT mode on AMD FX 8320 (8 core):
1. You need to set SetFilterMTMode("LWLibavVideoSource",3)

__________________
SVPflow motion interpolation

22nd March 2015, 15:43   #1032  |  Link
Groucho2004

Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
 Originally Posted by jones1913 I have indeed set "InvokeDistributor=0" because initially I had some problems get this to run with AVS+, so I tried several things.
FYI - The "InvokeDistributor()" setting is ignored in all Avisynth versions that don't implement "Distributor()".

22nd March 2015, 15:47   #1033  |  Link
Groucho2004

Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
 Originally Posted by chainik_svp 2. If you want to compare performance do it either in singe-threaded mode or with maximum CPU load (with 9+ threads). Your comparsion with 4 threads on 8-threads CPU shows nothing.
He used 4 threads in both cases, I don't see anything wrong with that.
Maximum CPU load or CPU load in general means very little. The most sensible thing would be to put CPU load and processing time in relation and compare that.

Last edited by Groucho2004; 22nd March 2015 at 15:53.

 22nd March 2015, 18:10 #1034  |  Link chainik_svp Registered User   Join Date: Mar 2012 Location: Saint-Petersburg Posts: 239 Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true . Those numbers can only be compared after normalization to 100% CPU load. AVS: 43.94 fps / 74% CPU load = 59.38 fps AVS+: 33.88 fps / 56% CPU load = 60.05 fps But such estimation gives too much error value. __________________ SVPflow motion interpolation
22nd March 2015, 18:41   #1035  |  Link
jones1913
random user

Join Date: May 2014
Location: #Neuland
Posts: 108
Quote:
 1. You need to set SetFilterMTMode("LWLibavVideoSource",3)
You're right, I wrongly assumed that the LWLibav filter is already specified in avs_plus_mt_modes.avsi file.
Though that has only a small effect on the result:
Code:
```SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("LWLibavVideoSource",3)
LWLibavVideoSource("sample.m2v")
QTGMC(Preset="Medium")
SelectEven()
Prefetch(4)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.741 | 121742 | 35.30
Memory usage (phys | virt):     350 | 366 MB
CPU usage (average):            55%

Time (elapsed):                 00:00:49.690```
Quote:
Well, "shows nothing" is maybe a little exaggerated. Also I have never stated to perform an absolute valid and technical perfect test.
Anyway I repeated the test with your suggestions:
Code:
```SetMTMode(3, 10)
LWLibavVideoSource("sample.m2v")
SetMTMode(2)
QTGMC(Preset="Medium")
SelectEven()

AVSMeter 1.9.8.0 (x86)
AviSynth 2.60, build:Feb 20 2015 [03:16:45] (2.6.0.5)
Active MT Mode: 2

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.636 | 167395 | 54.97
Memory usage (phys | virt):     634 | 674 MB
CPU usage (average):            96%

Time (elapsed):                 00:00:31.906```
Code:
```SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("LWLibavVideoSource",3)
LWLibavVideoSource("sample.m2v")
QTGMC(Preset="Medium")
SelectEven()
Prefetch(10)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      2.018 | 87337 | 48.62
Memory usage (phys | virt):     703 | 731 MB
CPU usage (average):            96%

Time (elapsed):                 00:00:36.077```
The gap becomes smaller, but the tendency remains the same.

Quote:
 Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true . Those numbers can only be compared after normalization to 100% CPU load. AVS: 43.94 fps / 74% CPU load = 59.38 fps AVS+: 33.88 fps / 56% CPU load = 60.05 fps But such estimation gives too much error value.
You posted this while I typed my above text and I have no idea if my results are meaningful, but I'll leave it as it is now...
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool

Last edited by jones1913; 22nd March 2015 at 18:43.

22nd March 2015, 19:15   #1036  |  Link
Groucho2004

Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
 Originally Posted by chainik_svp Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true
Quote:
 Originally Posted by ultim You enable MT by placing a single call to Prefetch(X) at the *end* of your script, where X is the number of threads to use.
Hm. So, ultim's explanation of "prefetch" is wrong in this post?

22nd March 2015, 20:18   #1037  |  Link
martin53
Registered User

Join Date: Mar 2007
Posts: 404
Quote:
 Originally Posted by ultim so you are writing your own avisynth plugin (cool!), and obviously one of the first things you have to do in your code is to include the avisynth header. But which one? ... what about avisynth+'s header? the headers of avisynth+ are up to date in every aspect and provide the greatest possible compatibility. By using avisynth+'s headers, applications and plugins can cleanly compile and run in 32-bits and 64-bits. It is 100% compatible to the latest 32-bit development on the old avisynth 2.6 project, while supporting all 64-bit binaries. And of course, you can use it regardless if you support multithreading or not. Furthermore and importantly, it is fully compatible to installations of the avisynth 2.6, avisynth-mt, avisynth64, and of course the avisynth+ projects, so your plugin/application will be able to run on any user's machine.
I'm sure there is reason in the things explained in that post, but:
Avisynth+'s header includes <avs/config.h>, <avs/capi.h> and <avs/types.h> which in turn include <avs/cpuid.h> and so on.
I feel this is not plugin writer friendly.
#include <...> in contrast to #include "..." makes success with the Filter SDK compiling instructions harder because it requires to further extend the standard library directories settings in the compiler.
Though I think I'd personally be able to manage that, and the files are on GitHub (easy to download), I still feel it's complicated and should not be like that. At quick glance I couldn't find what happened to the GetVarDef function in the avisynth+ header, either, so I'll preliminarily stick to the standard Avisynth 2.6 one.

22nd March 2015, 21:19   #1038  |  Link
chainik_svp
Registered User

Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
 Originally Posted by Groucho2004 Hm. So, ultim's explanation of "prefetch" is wrong in this post?
I'm just saying "4 threads" in AVS are different from "4 threads" in AVS+. Performance of AVS and AVS+ can't be compared in that way.
So there're only two reliable ways for comparsion: single thread performance and full system performance with 100% load.
__________________
SVPflow motion interpolation

22nd March 2015, 21:28   #1039  |  Link
chainik_svp
Registered User

Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
 Originally Posted by jones1913 Well, "shows nothing" is maybe a little exaggerated. Also I have never stated to perform an absolute valid and technical perfect test. Anyway I repeated the test with your suggestions:
yeah, and now it IS interesting thanks, I think now I should test the performance myself... (for the SVP usage )
__________________
SVPflow motion interpolation

23rd March 2015, 01:13   #1040  |  Link
qyot27
...?

Join Date: Nov 2005
Location: Florida
Posts: 1,301
Quote:
 Originally Posted by martin53 I'm sure there is reason in the things explained in that post, but: Avisynth+'s header includes , and which in turn include and so on. I feel this is not plugin writer friendly. #include <...> in contrast to #include "..." makes success with the Filter SDK compiling instructions harder because it requires to further extend the standard library directories settings in the compiler. Though I think I'd personally be able to manage that, and the files are on GitHub (easy to download), I still feel it's complicated and should not be like that.
https://github.com/AviSynth/AviSynthPlus/pull/54

Quote:
 At quick glance I couldn't find what happened to the GetVarDef function in the avisynth+ header, either, so I'll preliminarily stick to the standard Avisynth 2.6 one.
From https://github.com/AviSynth/AviSynthPlus/pull/53:

Code:
```partial-2   GetVarDef is one of a group of several similar functions that seem to be gone in
avsplus, so don't bother with it.```