Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 21st March 2015, 19:19   #1021  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Hey, I'm in luck today

I found out that the very first thing done by my cleaner function is the one that starts acting up.

Code:
SetFilterMTMode("DEFAULT_MT_MODE", 2)
DGSource("hotfuzztest.dgi")
Flux5framesT(th=2,chromamotion=true)
Prefetch(4)
and Flux5framesT along with what it needs:
Code:
function Flux5framesT(clip c, int "th", int "thC", bool "chromamotion")
{
th  = default(th,7)
thC = default(thC, (chromamotion == true) ? th : 0)
med  = c.TMedian2(chromamotion=chromamotion)
avg  = c.temporalsoften(2,th,thC,24,2)
medD = mt_makediff(c,med,U=chromamotion?3:1,V=chromamotion?3:1)
avgD = mt_makediff(c,avg,U=chromamotion?3:1,V=chromamotion?3:1)
DD   = mt_lutxy(medD,avgD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=chromamotion?3:1,V=chromamotion?3:1)
output = c.mt_makediff(DD,U=chromamotion?3:1,V=chromamotion?3:1)
return output
}

function TMedian2(clip c, bool "chromamotion")
{
Median2( c.selectevery(1,-2), c.selectevery(1,-1), c, c.selectevery(1,1), c.selectevery(1,2), chromamotion=chromamotion )
}

Function Median2(clip "input_1", clip "input_2", clip "input_3", clip "input_4", clip "input_5", string "chroma", bool "chromamotion")
{# median of 5 clips from Helpers.avs by G-force

chromamotion = default(chromamotion, true)
chroma = default(chroma, (chromamotion == true) ? "process" : "copy first") #default is "process". Alternates: "copy first" or "copy second"

#MEDIAN(i1,i3,i5)
Interleave(input_1,input_3,input_5)
chroma == "process" ? Clense() : Clense(grey=true)
#chroma == "process" ? MedianBlurT(0,0,0,1).merge(last,0.5).MedianBlurT(0,0,0,1) : MedianBlurT(0,-1,-1,1).merge(last,0.5).MedianBlurT(0,-1,-1,1)
m1 = selectevery(3,1)

#MAX(MIN(i1,i3,i5),i2)
m2  = input_1.MT_Logic(input_3,"min",chroma=chroma).MT_Logic(input_5,"min",chroma=chroma).MT_Logic(input_2,"max",chroma=chroma)

#MIN(MAX(i1,i3,i5),i4)
m3  = input_1.MT_Logic(input_3,"max",chroma=chroma).MT_Logic(input_5,"max",chroma=chroma).MT_Logic(input_4,"min",chroma=chroma)

Interleave(m1,m2,m3)
chroma == "process" ? Clense() : Clense(grey=true)
#chroma == "process" ? MedianBlurT(0,0,0,1).merge(last,0.5).MedianBlurT(0,0,0,1) : MedianBlurT(0,-1,-1,1).merge(last,0.5).MedianBlurT(0,-1,-1,1)
selectevery(3,1)

chroma == "copy first" ? last.MergeChroma(input_1) : chroma == "copy second" ? last.MergeChroma(input_2) : last

Return(last)
}
I think these two plugins are the only ones that are needed: https://drive.google.com/file/d/0Bze...ew?usp=sharing


EDIT: when replacing the two "chroma ==" lines with the MedianBlurT version in the Median2 function, I just get a crash when opening the script.

EDIT2: In fact, in AVS+ the function doesn't seem to do anything. Comparing it with the original clip doesn't show any differences, with SEt's MT build the changes are obvious.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...

Last edited by Boulder; 21st March 2015 at 19:34.
Boulder is offline  
Old 21st March 2015, 20:54   #1022  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by Boulder View Post
I was thinking more on the lines of finding out if there are possible problems in memory handling as ultim said he's looking to make the cache handling more robust. Then again, I seem to get the script to stall even with SetMemoryMax(256) and two threads.
Yes the caches need some fine tuning. Currently they work great until memory starts running out.

It would certainly help a lot to be able to test with your scripts. If you use a lot of helper scripts, just post them too If you don't want to make them public, maybe you can PM or e-mail them to me?

EDIT: Oh, you made a new post while I wrote my own. Thanks, I'll check it out.
__________________
AviSynth+

Last edited by ultim; 21st March 2015 at 20:57.
ultim is offline  
Old 22nd March 2015, 06:50   #1023  |  Link
colours
Registered User
 
colours's Avatar
 
Join Date: Mar 2014
Posts: 308
Quote:
Originally Posted by Boulder View Post
I found out that the very first thing done by my cleaner function is the one that starts acting up.

Code:
reallylongscript
I know this is not very relevant, but I'm seeing a whole lot of optimisation potential there. Not tested, but should produce identical results and be faster.

Code:
function Flux5framesT(clip c,int "th",int "thC",bool "chromamotion")
{
	th  = default(th, 7)
	thC = default(thC, chromamotion ? th : 0)
	med = chromamotion ? ytouv(c.utoy8().median5t(), c.vtoy8().median5t(), c.median5t()) : c.median5t().mergechroma(c)
	avg = c.temporalsoften(2, th, thC, 24, 2)
	output = interleave(c, med, avg).clense(grey=!chromamotion).selectevery(3,1)
	return output
}

function median5t(clip src)
{ # from here
	function min(clip a, clip b) {return mt_logic(a, b, mode="min")}
	function max(clip a, clip b) {return mt_logic(a, b, mode="max")}
	src
	last + trim(framecount()-1,-1).loop(5)
	bcmin = min(SelectEvery(2, -1), SelectEvery(2, 0))
	bcmax = max(SelectEvery(2, -1), SelectEvery(2, 0))
	demin = bcmin.SelectEvery(1, 1)
	demax = bcmax.SelectEvery(1, 1)
	x = max(bcmin, demin)
	y = min(bcmax, demax)
	a = SelectEvery(2, -2)
	f = SelectEvery(2, 3)
	Interleave(a, x, y, f).Clense(grey=true).SelectEvery(4, 1, 2)
	trim(0,length=src.framecount())
}
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!
colours is offline  
Old 22nd March 2015, 10:06   #1024  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Thanks for those, I'll test them and replace the old ones if they are indeed faster The old ones have sometimes been posted here on this forum and I've borrowed them as a small pre-cleaner function to the MC'd denoiser.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 22nd March 2015, 11:47   #1025  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Flux5FramesT needed just the line "chromamotion = default(chromamotion, true)" so I added that one since I very rarely disable chroma motion detection.

However, the initial problem still remains, a test with AVSMeter stalls the process at some point. I'm also getting CPU usage below 25% (1 core) even with four threads, maybe that's something to look at (caching again?)
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 22nd March 2015, 11:47   #1026  |  Link
jones1913
random user
 
Join Date: May 2014
Location: #Neuland
Posts: 116
Quote:
Originally Posted by noee View Post
I've been testing with AVS+/MT mode w/QTGMC. So far, I haven't checked numbers against SETs 2.6RC1, but fwiw, in 64bit version, AVS+ will complete an encode that fails with the older (JoshyD?) 64-bit build.

Have you tried yet? If so, what kind of numbers do you see?
Do you mean that you run QTGMC with 64 bit AVS? Until now I assumed that not all needed plugins are available as 64 bit version.

Here are my results with QTGMC in MT mode on AMD FX 8320 (8 core):

AVS 2.6:
Code:
SetMTMode(3, 4)
LWLibavVideoSource("sample.m2v") 
SetMTMode(2)
QTGMC(Preset="Medium")
SelectEven()
Distributor()

AVSMeter 1.9.8.0 (x86)
AviSynth 2.60, build:Feb 20 2015 [03:16:45] (2.6.0.5)
Active MT Mode: 2

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      5.680 | 91306 | 43.94
Memory usage (phys | virt):     585 | 604 MB
Thread count:                   65
CPU usage (average):            74%

Time (elapsed):                 00:00:39.919
AVS+:
Code:
SetFilterMTMode("DEFAULT_MT_MODE",2)
LWLibavVideoSource("sample.m2v") 
QTGMC(Preset="Medium")
SelectEven()
Prefetch(4)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.676 | 114785 | 33.88
Memory usage (phys | virt):     398 | 414 MB
Thread count:                   104
CPU usage (average):            56%

Time (elapsed):                 00:00:51.768
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool
jones1913 is offline  
Old 22nd March 2015, 12:43   #1027  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by jones1913 View Post
Do you mean that you run QTGMC with 64 bit AVS? Until now I assumed that not all needed plugins are available as 64 bit version.
I've got it working several years ago with AVIsynth x64. I don't recall it being much faster in 64-bit though. It was a giant pain to find x64 versions of all the plugins.
Stereodude is offline  
Old 22nd March 2015, 13:03   #1028  |  Link
noee
Registered User
 
Join Date: Jan 2007
Posts: 530
Quote:
Originally Posted by Stereodude View Post
I've got it working several years ago with AVIsynth x64. I don't recall it being much faster in 64-bit though. It was a giant pain to find x64 versions of all the plugins.
Yes, that seems to be the case. My scripts with QTGMC are quite simple and when it would work in 64bit, it was about 3% faster than 32bit MT (SET).
__________________
Win7Ult || RX560/4G || Ryzen 5
noee is offline  
Old 22nd March 2015, 13:05   #1029  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by jones1913 View Post
Code:
Distributor()
Don't put that in the script unless you specified "InvokeDistributor=0" in AVSMeter.ini.
That goes for pretty much every application that uses AVISynth(MT) except ffdshow and some piping tools that don't call Distributor() internally.
Groucho2004 is offline  
Old 22nd March 2015, 13:46   #1030  |  Link
jones1913
random user
 
Join Date: May 2014
Location: #Neuland
Posts: 116
Quote:
Don't put that in the script unless you specified "InvokeDistributor=0" in AVSMeter.ini.
I have indeed set "InvokeDistributor=0" because initially I had some problems get this to run with AVS+, so I tried several things.

Quote:
That goes for pretty much every application that uses AVISynth(MT) except ffdshow and some piping tools that don't call Distributor() internally.
OK I didn't know that, thanks.
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool
jones1913 is offline  
Old 22nd March 2015, 14:17   #1031  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
Originally Posted by jones1913
Here are my results with QTGMC in MT mode on AMD FX 8320 (8 core):
1. You need to set SetFilterMTMode("LWLibavVideoSource",3)

2. If you want to compare performance do it either in singe-threaded mode or with maximum CPU load (with 9+ threads). Your comparsion with 4 threads on 8-threads CPU shows nothing.
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 22nd March 2015, 15:43   #1032  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by jones1913 View Post
I have indeed set "InvokeDistributor=0" because initially I had some problems get this to run with AVS+, so I tried several things.
FYI - The "InvokeDistributor()" setting is ignored in all Avisynth versions that don't implement "Distributor()".
Groucho2004 is offline  
Old 22nd March 2015, 15:47   #1033  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by chainik_svp View Post
2. If you want to compare performance do it either in singe-threaded mode or with maximum CPU load (with 9+ threads). Your comparsion with 4 threads on 8-threads CPU shows nothing.
He used 4 threads in both cases, I don't see anything wrong with that.
Maximum CPU load or CPU load in general means very little. The most sensible thing would be to put CPU load and processing time in relation and compare that.

Last edited by Groucho2004; 22nd March 2015 at 15:53.
Groucho2004 is offline  
Old 22nd March 2015, 18:10   #1034  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true .

Those numbers can only be compared after normalization to 100% CPU load.
AVS: 43.94 fps / 74% CPU load = 59.38 fps
AVS+: 33.88 fps / 56% CPU load = 60.05 fps

But such estimation gives too much error value.
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 22nd March 2015, 18:41   #1035  |  Link
jones1913
random user
 
Join Date: May 2014
Location: #Neuland
Posts: 116
Quote:
1. You need to set SetFilterMTMode("LWLibavVideoSource",3)
You're right, I wrongly assumed that the LWLibav filter is already specified in avs_plus_mt_modes.avsi file.
Though that has only a small effect on the result:
Code:
SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("LWLibavVideoSource",3)
LWLibavVideoSource("sample.m2v") 
QTGMC(Preset="Medium")
SelectEven()
Prefetch(4)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.741 | 121742 | 35.30
Memory usage (phys | virt):     350 | 366 MB
Thread count:                   68
CPU usage (average):            55%

Time (elapsed):                 00:00:49.690
Quote:
2. If you want to compare performance do it either in singe-threaded mode or with maximum CPU load (with 9+ threads). Your comparsion with 4 threads on 8-threads CPU shows nothing.
Well, "shows nothing" is maybe a little exaggerated. Also I have never stated to perform an absolute valid and technical perfect test.
Anyway I repeated the test with your suggestions:
Code:
SetMTMode(3, 10)
LWLibavVideoSource("sample.m2v") 
SetMTMode(2)
QTGMC(Preset="Medium")
SelectEven()

AVSMeter 1.9.8.0 (x86)
AviSynth 2.60, build:Feb 20 2015 [03:16:45] (2.6.0.5)
Active MT Mode: 2

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      3.636 | 167395 | 54.97
Memory usage (phys | virt):     634 | 674 MB
Thread count:                   107
CPU usage (average):            96%

Time (elapsed):                 00:00:31.906
Code:
SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("LWLibavVideoSource",3)
LWLibavVideoSource("sample.m2v") 
QTGMC(Preset="Medium")
SelectEven()
Prefetch(10)

AVSMeter 1.9.8.0 (x86)
AviSynth+ 0.1 (r1779, MT, i386) (0.1.0.0)

Number of frames:                 1754
Length (hh:mm:ss.ms):     00:01:10.160
Frame width:                       720
Frame height:                      576
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               1754 (0 - 1753)
FPS (min | max | average):      2.018 | 87337 | 48.62
Memory usage (phys | virt):     703 | 731 MB
Thread count:                   123
CPU usage (average):            96%

Time (elapsed):                 00:00:36.077
The gap becomes smaller, but the tendency remains the same.

Quote:
Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true .

Those numbers can only be compared after normalization to 100% CPU load.
AVS: 43.94 fps / 74% CPU load = 59.38 fps
AVS+: 33.88 fps / 56% CPU load = 60.05 fps

But such estimation gives too much error value.
You posted this while I typed my above text and I have no idea if my results are meaningful, but I'll leave it as it is now...
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool

Last edited by jones1913; 22nd March 2015 at 18:43.
jones1913 is offline  
Old 22nd March 2015, 19:15   #1036  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by chainik_svp View Post
Because AVS and AVS+ have completely different thread schedulers and "4" is not a "number of threads" but just one of input parameters to the scheduler. For AVS+ official point of view is given in the function name "Prefetch" - "4" is number of frames calculated in advance. Which is also not true
Quote:
Originally Posted by ultim View Post
You enable MT by placing a single call to Prefetch(X) at the *end* of your script, where X is the number of threads to use.
Hm. So, ultim's explanation of "prefetch" is wrong in this post?
Groucho2004 is offline  
Old 22nd March 2015, 20:18   #1037  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by ultim View Post
so you are writing your own avisynth plugin (cool!), and obviously one of the first things you have to do in your code is to include the avisynth header. But which one? ...
what about avisynth+'s header?
the headers of avisynth+ are up to date in every aspect and provide the greatest possible compatibility. By using avisynth+'s headers, applications and plugins can cleanly compile and run in 32-bits and 64-bits. It is 100% compatible to the latest 32-bit development on the old avisynth 2.6 project, while supporting all 64-bit binaries. And of course, you can use it regardless if you support multithreading or not. Furthermore and importantly, it is fully compatible to installations of the avisynth 2.6, avisynth-mt, avisynth64, and of course the avisynth+ projects, so your plugin/application will be able to run on any user's machine.
I'm sure there is reason in the things explained in that post, but:
Avisynth+'s header includes <avs/config.h>, <avs/capi.h> and <avs/types.h> which in turn include <avs/cpuid.h> and so on.
I feel this is not plugin writer friendly.
#include <...> in contrast to #include "..." makes success with the Filter SDK compiling instructions harder because it requires to further extend the standard library directories settings in the compiler.
Though I think I'd personally be able to manage that, and the files are on GitHub (easy to download), I still feel it's complicated and should not be like that. At quick glance I couldn't find what happened to the GetVarDef function in the avisynth+ header, either, so I'll preliminarily stick to the standard Avisynth 2.6 one.
martin53 is offline  
Old 22nd March 2015, 21:19   #1038  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
Originally Posted by Groucho2004 View Post
Hm. So, ultim's explanation of "prefetch" is wrong in this post?
I'm just saying "4 threads" in AVS are different from "4 threads" in AVS+. Performance of AVS and AVS+ can't be compared in that way.
So there're only two reliable ways for comparsion: single thread performance and full system performance with 100% load.
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 22nd March 2015, 21:28   #1039  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
Quote:
Originally Posted by jones1913
Well, "shows nothing" is maybe a little exaggerated. Also I have never stated to perform an absolute valid and technical perfect test.
Anyway I repeated the test with your suggestions:
yeah, and now it IS interesting thanks, I think now I should test the performance myself... (for the SVP usage )
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 23rd March 2015, 01:13   #1040  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
Quote:
Originally Posted by martin53 View Post
I'm sure there is reason in the things explained in that post, but:
Avisynth+'s header includes <avs/config.h>, <avs/capi.h> and <avs/types.h> which in turn include <avs/cpuid.h> and so on.
I feel this is not plugin writer friendly.
#include <...> in contrast to #include "..." makes success with the Filter SDK compiling instructions harder because it requires to further extend the standard library directories settings in the compiler.
Though I think I'd personally be able to manage that, and the files are on GitHub (easy to download), I still feel it's complicated and should not be like that.
https://github.com/AviSynth/AviSynthPlus/pull/54

Quote:
At quick glance I couldn't find what happened to the GetVarDef function in the avisynth+ header, either, so I'll preliminarily stick to the standard Avisynth 2.6 one.
From https://github.com/AviSynth/AviSynthPlus/pull/53:

Code:
partial-2   GetVarDef is one of a group of several similar functions that seem to be gone in
        avsplus, so don't bother with it.
qyot27 is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:54.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.