Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th April 2018, 15:41   #61  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by jpsdr View Post
The script are provided, so, if looking at them it's impossible to say what is benchmarked, i indeed don't know what to do more.
That's not what I mean. You're arguing that your setAffinity thing is improving performance by improving the CPU cache hit ratio. You then provide a measurement that is supposed to be supporting this argument. I can't say what your benchmark is actually measuring since I don't know where the bottleneck actually is, but it's definitely not CPU cache hit ratio, and since your implementation appears to give drastically different results for other people I suspect the implementation doesn't do what you think it does. Even if it did improve performance consistently, the measurement would still not be saying anything about CPU cache hit ratio.

Quote:
Originally Posted by jpsdr View Post
About cache, i'm just saying that if you have 8 physical CPUs with 8 threads workings each one on 1/8 of 1Mb frame and each thread on a different CPU, there is more chances that the working memory zone of each threads will totaly fit and stay within the cache during the whole process, than if you have 8 threads working each one on a full 1Mb frame.
No more, no less.
Again, why do you think this is advantageous? Consider a trivial case: we're downscaling an image 2x in both directions with pointresize. Each destination pixel will only sample two input pixels per pass (horizontal/vertical) and they never get reused after that. Pretty much all you need is to fit a single scanline in cache.
TheFluff is offline   Reply With Quote
Old 4th April 2018, 16:48   #62  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by jpsdr View Post
@Atak_Snajpera

Can you provide yours results with both true/false for just the following script :

Code:
Colorbars(width=1920*2,height=1080*2,pixel_type="yv12").killaudio().assumefps(25,1).trim(0,9999)
Spline36ResizeMT(1920,1080,SetAffinity=true)
No need to bother with pictures, just paste the [Runtime info] from the log file, it should be easier and faster for you.
true
Code:
AVSMeter 2.2.6 (x64)
AviSynth+ 0.1 (r2580, MT, x86_64) (0.1.0.0)
Loading script...

Number of frames:                10000
Length (hh:mm:ss.ms):     00:06:40.000
Frame width:                      1920
Frame height:                     1080
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               10000 (0 - 9999)
FPS (min | max | average):      140.6 | 408.9 | 342.0
Memory usage (phys | virt):     35 | 30 MiB
Thread count:                   33
CPU usage (average):            81%

Time (elapsed):                 00:00:29.236
false
Code:
AVSMeter 2.2.6 (x64)
AviSynth+ 0.1 (r2580, MT, x86_64) (0.1.0.0)
Loading script...

Number of frames:                10000
Length (hh:mm:ss.ms):     00:06:40.000
Frame width:                      1920
Frame height:                     1080
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frames processed:               10000 (0 - 9999)
FPS (min | max | average):      214.8 | 412.9 | 362.1
Memory usage (phys | virt):     35 | 30 MiB
Thread count:                   33
CPU usage (average):            85%

Time (elapsed):                 00:00:27.614
It works with colorbars. I think You should start testing with some real footage like me. (3840x2160 HDR + crop + resize).
Summary. Messing with affinity makes zero sense.

Last edited by Atak_Snajpera; 4th April 2018 at 16:52.
Atak_Snajpera is offline   Reply With Quote
Old 4th April 2018, 17:02   #63  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Something is strange.
You have here, as expected, 33 threads : 1 + 16 from avs+ + 16 from internal multi-threading.
In your previous posts, you have 49 threads, so 16 more. They could be explained if you have added prefetch(16) in your script, but it's not the case. I'm wondering where they are comming from...
I'll make some tests at home with a file.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 4th April 2018, 17:32   #64  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
You do realize that ffms decoder spawns additional threads? Those extra 16 threads are basically from decoder.

Code:
#VideoSource
LoadPlugin("C:\Users\Dave\Documents\Delphi_Projects\RipBot264\_Compiled\Tools\AviSynth plugins\ffms\ffms_latest\x64\ffms2.dll")
video=FFVideoSource("E:\_Video_Samples\mkv\Passengers_2016_4K.mkv",cachefile = "C:\Temp\RipBot264temp\job1\Passengers_2016_4K.mkv.ffindex")

Last edited by Atak_Snajpera; 4th April 2018 at 17:35.
Atak_Snajpera is offline   Reply With Quote
Old 4th April 2018, 18:42   #65  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Yes, of course !

And i have also the same things in these results :

Code:
[Clip info]
Number of frames:                13200
Length (hh:mm:ss.ms):     00:09:10.550
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       YV12

[Runtime info]
Frames processed:               13200 (0 - 13199)
FPS (min | max | average):      21.43 | 192.1 | 153.3
Memory usage (phys | virt):     92 | 102 MiB
Thread count:                   61
CPU usage (average):            46%

Time (elapsed):                 00:01:26.106

[Script]
a=AviSource("Dark_Crystal_HDR.avi",False,"YV12")
b=a+a+a+a+a+a+a+a+a+a+a
c=b+b+b+b+b+b+b+b+b+b+b+b
Spline36ResizeMT(c,1920,1080,SetAffinity=true,threads=0)
Code:
[Clip info]
Number of frames:                13200
Length (hh:mm:ss.ms):     00:09:10.550
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       YV12

[Runtime info]
Frames processed:               13200 (0 - 13199)
FPS (min | max | average):      21.30 | 185.2 | 149.0
Memory usage (phys | virt):     92 | 102 MiB
Thread count:                   61
CPU usage (average):            40%

Time (elapsed):                 00:01:28.569

[Script]
#SetMemoryMax(192)
a=AviSource("Dark_Crystal_HDR.avi",False,"YV12")
b=a+a+a+a+a+a+a+a+a+a+a
c=b+b+b+b+b+b+b+b+b+b+b+b
Spline36ResizeMT(c,1920,1080,SetAffinity=false,threads=0)
Code:
[Clip info]
Number of frames:                13200
Length (hh:mm:ss.ms):     00:09:10.550
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       YV12

[Runtime info]
Frames processed:               13200 (0 - 13199)
FPS (min | max | average):      47.03 | 186.4 | 173.7
Memory usage (phys | virt):     306 | 478 MiB
Thread count:                   49
CPU usage (average):            16%

Time (elapsed):                 00:01:15.980

[Script]
DGSource("DCrystal.dgi",i420=false,deinterlace=0,fieldop=0).trim(0,13199)
Spline36ResizeMT(1920,1080,SetAffinity=true,threads=0)
Code:
[Clip info]
Number of frames:                13200
Length (hh:mm:ss.ms):     00:09:10.550
Frame width:                      1920
Frame height:                     1080
Framerate:                      23.976 (24000/1001)
Colorspace:                       YV12

[Runtime info]
Frames processed:               13200 (0 - 13199)
FPS (min | max | average):      54.67 | 180.2 | 172.9
Memory usage (phys | virt):     306 | 478 MiB
Thread count:                   49
CPU usage (average):            14%

Time (elapsed):                 00:01:16.342

[Script]
DGSource("DCrystal.dgi",i420=false,deinterlace=0,fieldop=0).trim(0,13199)
Spline36ResizeMT(1920,1080,SetAffinity=false,threads=0)
As i've said, it seems that sometimes you need to tune for yourself, what's good for one, is not always good for another.
__________________
My github.

Last edited by jpsdr; 4th April 2018 at 18:44.
jpsdr is offline   Reply With Quote
Old 4th April 2018, 18:54   #66  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Quote:
Originally Posted by TheFluff View Post
Again, why do you think this is advantageous?
I'll clarify my thought : It may be, not it will be. Also, don't focuss only on resampler, maybe there is different filters on which it could be more efficient.
And again, if what is good for me is not for you, there is no issue, because you can change it. It's not like it's forced and you don't have the choice.
As they are my builds, i set the default settings to what give me the best results for my use case, so i play selfish, to not have to adjust for myself....
__________________
My github.
jpsdr is offline   Reply With Quote
Old 4th April 2018, 19:05   #67  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
As i've said, it seems that sometimes you need to tune for yourself, what's good for one, is not always good for another.
SetAffinity should be disabled by default. There is more trouble than benefit. Speed increase in your example is less than 3%.

Last edited by Atak_Snajpera; 4th April 2018 at 19:08.
Atak_Snajpera is offline   Reply With Quote
Old 4th April 2018, 19:19   #68  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Sorry, but for now, i'll keep it this way.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 5th April 2018, 04:42   #69  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
after I update for the last resample I note there are some slow and after I see Atak_Snajpera posts here I decided to test with avsmeter


SetAffinity test

old cpu (2010) on old windows


new cpu (2014 maybe not really new) on new windows


both said there are problem in SetAffinity in last update, the previous version was fine https://forum.doom9.org/showpost.php...0&postcount=41
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 5th April 2018, 09:02   #70  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Argh... Two against one... This is begining to be harder. Why am i the only one where it works differently...
I know !!! It's a conspiracy ! Ah... I knew the earth was indeed flat !
Out of curiousity, what is your script ?
__________________
My github.

Last edited by jpsdr; 5th April 2018 at 09:06.
jpsdr is offline   Reply With Quote
Old 5th April 2018, 09:24   #71  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by jpsdr View Post
Why am i the only one where it works differently...
Maybe your Broadwell 10/20 core CPU behaves differently compared to the CPUs that we mere mortals use?
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 5th April 2018 at 10:59.
Groucho2004 is offline   Reply With Quote
Old 5th April 2018, 09:36   #72  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Quote:
Originally Posted by Groucho2004 View Post
Maybe your Broadwell 10/20 core CPU behaves differently compared to the CPUs that we mere mortals use?
Ah... That's it of course... Poor low mortals...
__________________
My github.
jpsdr is offline   Reply With Quote
Old 5th April 2018, 12:41   #73  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
I can also confirm that previous version 2.0.5 worked ok (or was bugged and was always false) with default setaffinity=true.
SetAffinity is useless because:
1) Unpredictable performance for many users.
2) Extremely tiny speed boost for users with specific CPU (Broadwell). Furthermore that extra 3% will be most likely further "eaten" by other filter or procces (x264/x265)
3) has to be disabled when prefetch is used in script

SetAffinity=true as default setting is NOT optimal. Period!

Last edited by Atak_Snajpera; 5th April 2018 at 12:44.
Atak_Snajpera is offline   Reply With Quote
Old 5th April 2018, 14:30   #74  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by jpsdr View Post
Argh... Two against one... This is begining to be harder. Why am i the only one where it works differently...
I know !!! It's a conspiracy ! Ah... I knew the earth was indeed flat !
Out of curiousity, what is your script ?
I used this

Code:
Colorbars(width=1920*2,height=1080*2,pixel_type="yv12").killaudio().assumefps(25,1).trim(0,9999)
Spline36ResizeMT(1920,1080,SetAffinity=true)
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 7th April 2018, 12:45   #75  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
New version, see first post, updated also the Multi-treading text part.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 1st June 2018, 10:05   #76  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
New version, see first post.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 2nd June 2018, 07:43   #77  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Updated, thank you!
FranceBB is offline   Reply With Quote
Old 14th August 2018, 17:51   #78  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
New version, add HDRTools, see first post.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 29th December 2018, 12:37   #79  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Slipstreamed a new build/file. There is no change in the code, so it's not a new version.
I've realised that i didn't put the documentation of the plugins in this package, and if you wanted the documentation you also needed to download each plugin version, which is not very convenient (and not smart from my part). So, i've added in the release file the documentation of each plugin.
I took the opportunuity to make new builds with the last Visual Studio release, and also put back the Intel Compiler builds.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 1st June 2019, 12:00   #80  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
New version, see first post.
__________________
My github.
jpsdr is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.