Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 1st September 2011, 17:21   #221  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by fabje View Post
I will do this as soon as the person who has the same system is back from his vacation.
indeed


You have SSD... does he? (SSD drivers are IMHO beta still and may be causing a conflict on your box. disable driver and see)

are the CPU/RAM speeds forced in the BIOS or not?


Quote:
Originally Posted by fabje View Post
Also a weird thing is that when I do a re-encode with the x264 codec of a 1920 x 1080 recording to a 1280 x 720 file I get the same speeds as my friend.
well what does that tell you? It tells me you have an I/O bottleneck that he does not since x264 is so CPU heavy any I/O throughput problem is masked. This why i want the baseline with just

MPEG2Source("E:\Recordings\test.d2v")
(No encoding just that to test throughput)

because any decent box can read 40 frames a second in but when you get up to 400 FPS well.... that is a lot of data.


You may have a driver installed he does not that is slowing you down. A good example is most all anti virus. The default is for them to scan a file on OPEN READ WRITE and CLOSE. this will slow you way down and you should only need to scan once on OPEN and once on CLOSE. if you let it scan during read/write your I/O will be cut in half or more.

no need to wait for your friend if you make a ram disk and test from that. (note if you can't get at least 1 gig second read from ramdisk your anti virus or some other driver is slowing it down)


anyway just more testing needed and we will know enough... good luck

Last edited by Zep; 1st September 2011 at 17:26.
Zep is offline   Reply With Quote
Old 1st September 2011, 17:56   #222  |  Link
fabje
Registered User
 
Join Date: Mar 2010
Posts: 98
Quote:
Originally Posted by Zep View Post
indeed


You have SSD... does he? (SSD drivers are IMHO beta still and may be causing a conflict on your box. disable driver and see)

are the CPU/RAM speeds forced in the BIOS or not?




well what does that tell you? It tells me you have an I/O bottleneck that he does not since x264 is so CPU heavy any I/O throughput problem is masked. This why i want the baseline with just

MPEG2Source("E:\Recordings\test.d2v")
(No encoding just that to test throughput)

because any decent box can read 40 frames a second in but when you get up to 400 FPS well.... that is a lot of data.


You may have a driver installed he does not that is slowing you down. A good example is most all anti virus. The default is for them to scan a file on OPEN READ WRITE and CLOSE. this will slow you way down and you should only need to scan once on OPEN and once on CLOSE. if you let it scan during read/write your I/O will be cut in half or more.

no need to wait for your friend if you make a ram disk and test from that. (note if you can't get at least 1 gig second read from ramdisk your anti virus or some other driver is slowing it down)


anyway just more testing needed and we will know enough... good luck
Yes he has a SSD but he doesn't use that for encoding.

And a little correction my friend is using 2 WD Blacks in RAID0.

My CPU/RAM speeds are indeed forced in the BIOS. So all that energy saving crap is disabled.

When I'm running the baseline I also get around 250 fps.
My anti virus software isn't the problem because I had this problem already when I just had a clean install of Windows 7 with only the drivers installed for my motherboard and graphic card. And I used the same drivers as my friend did.
__________________
CPU: i7 3930K @ 4.5Ghz MEM: 16GB GPU:NVIDIA GeForce GTX680 OS: Windows 7 64Bit

Last edited by fabje; 1st September 2011 at 22:13.
fabje is offline   Reply With Quote
Old 5th September 2011, 18:11   #223  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by fabje View Post
Yes he has a SSD but he doesn't use that for encoding.

And a little correction my friend is using 2 WD Blacks in RAID0.

well that little correction is huge. I'm not saying it is the only problem but from what you said so far it is the main difference i see. Black drives get 40% more sustained throughput on reads than greens.

he gets about 400 you get about 250 well.... there you go lol


I say again. test from a ram disk to make sure.
Zep is offline   Reply With Quote
Old 5th September 2011, 18:37   #224  |  Link
fabje
Registered User
 
Join Date: Mar 2010
Posts: 98
Quote:
Originally Posted by Zep View Post
well that little correction is huge. I'm not saying it is the only problem but from what you said so far it is the main difference i see. Black drives get 40% more sustained throughput on reads than greens.

he gets about 400 you get about 250 well.... there you go lol


I say again. test from a ram disk to make sure.
And how can I test from a RAM disk?
__________________
CPU: i7 3930K @ 4.5Ghz MEM: 16GB GPU:NVIDIA GeForce GTX680 OS: Windows 7 64Bit
fabje is offline   Reply With Quote
Old 5th September 2011, 19:17   #225  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by fabje View Post
And how can I test from a RAM disk?
I use this and it works great (yes it is free if only 4 gig ram disk or smaller)

http://memory.dataram.com/products-a...ftware/ramdisk

then move a .ts or .mpg onto the ram disk and run your script so that it points to the source file on that ram disk. if it's an HD throughput wall you will know real quick. (assuming you are no where near 100% CPU etc... I do not recall if you mentioned that or not)
Zep is offline   Reply With Quote
Old 7th September 2011, 20:13   #226  |  Link
fabje
Registered User
 
Join Date: Mar 2010
Posts: 98
Quote:
Originally Posted by Zep View Post
I use this and it works great (yes it is free if only 4 gig ram disk or smaller)

http://memory.dataram.com/products-a...ftware/ramdisk

then move a .ts or .mpg onto the ram disk and run your script so that it points to the source file on that ram disk. if it's an HD throughput wall you will know real quick. (assuming you are no where near 100% CPU etc... I do not recall if you mentioned that or not)
Well I did the test and I got the same results.
So seems like my HDD isn't the problem.

My CPU has an usage of about 50%.
I also did a check of how many threads my pc and the pc of my friend have and we are both around 880 threads.

And I did a quick HDD benchmark and this are the results:
Transfer Rate:
Minimum: 114.7MB/sec
Maximum: 156.3MB/sec
Average: 149.8MB/sec

Access Rate: 10.9ms
Burst Rate: 1295.9 MB/sec

And yes his average is higher but if I check my SSD, the SSD is faster but still this didn't changed a bit.
__________________
CPU: i7 3930K @ 4.5Ghz MEM: 16GB GPU:NVIDIA GeForce GTX680 OS: Windows 7 64Bit

Last edited by fabje; 7th September 2011 at 20:35.
fabje is offline   Reply With Quote
Old 7th September 2011, 22:06   #227  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by fabje View Post

My CPU has an usage of about 50%.

so when your friend runs just MPEG2Source("E:\Recordings\test.d2v")

no encoding just a pass through FPS test he gets 400 and you get 250? You both are using same versions of software? Same bios version? same drivers versions? etc.... get process explorer and look at the avisynth threads. Compare your boxes and you should be able to see where the bottleneck is. you can even click on a single thread and see the I/O rates and much more. yeah it is time to go lower level lol


good luck
Zep is offline   Reply With Quote
Old 8th September 2011, 06:56   #228  |  Link
fabje
Registered User
 
Join Date: Mar 2010
Posts: 98
Quote:
Originally Posted by Zep View Post
so when your friend runs just MPEG2Source("E:\Recordings\test.d2v")

no encoding just a pass through FPS test he gets 400 and you get 250? You both are using same versions of software? Same bios version? same drivers versions? etc.... get process explorer and look at the avisynth threads. Compare your boxes and you should be able to see where the bottleneck is. you can even click on a single thread and see the I/O rates and much more. yeah it is time to go lower level lol


good luck
No my friend gets 400fps when he is using this script:
SetMTMode(5,3)
MPEG2Source("E:\Recordings\recording.d2v")
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
SetMTMode(2)
LeakKernelDeint(order=1,threshold=8)
Crop(2,2,-2,-2)
BicubicResize(656,368,0,0.5)
Trim(9992,46915)

And yes our software version etc is the same.
The only thing that is really different is that my cpu is overclocked and his pc isn't. So I thought maybe it was an setting in my bios, but even with everything in there on default. My speeds are getting slower then what I have right now.
__________________
CPU: i7 3930K @ 4.5Ghz MEM: 16GB GPU:NVIDIA GeForce GTX680 OS: Windows 7 64Bit
fabje is offline   Reply With Quote
Old 8th September 2011, 10:59   #229  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
Quote:
Originally Posted by fabje View Post
... The only thing that is really different is that my cpu is overclocked and his pc isn't. ....
Overclocking is a very black art. Try running the tests with the exact same settings as your friend.

One possible pitfall is memory access pattern, it is possible to only get down to half the effective memory speed when comparing two CPU speeds that are almost the same This happens when the slightly faster speed requires stepping down to a much crappier memory access pattern.

Also some memory controllers drop to a lower speed as you add more than a certain number of memory modules (bus loading problems). We had 2 supposedly identical servers shipped, 1 full of 1GB memory sticks the other with nice 2GB memory sticks .....
IanB is offline   Reply With Quote
Old 8th September 2011, 11:23   #230  |  Link
fabje
Registered User
 
Join Date: Mar 2010
Posts: 98
Quote:
Originally Posted by IanB View Post
Overclocking is a very black art. Try running the tests with the exact same settings as your friend.

One possible pitfall is memory access pattern, it is possible to only get down to half the effective memory speed when comparing two CPU speeds that are almost the same This happens when the slightly faster speed requires stepping down to a much crappier memory access pattern.

Also some memory controllers drop to a lower speed as you add more than a certain number of memory modules (bus loading problems). We had 2 supposedly identical servers shipped, 1 full of 1GB memory sticks the other with nice 2GB memory sticks .....
I did try it with the exact same setting as my friend.
The only thing I could try is to test it with the same memory settings.

I also did a process test and this are the results when i'm encoding something on my pc.



And this is a screenshot of the process explorer on the pc of my friend.
__________________
CPU: i7 3930K @ 4.5Ghz MEM: 16GB GPU:NVIDIA GeForce GTX680 OS: Windows 7 64Bit

Last edited by fabje; 8th September 2011 at 17:58.
fabje is offline   Reply With Quote
Old 9th September 2011, 14:33   #231  |  Link
matfra
Registered User
 
Join Date: Jul 2009
Posts: 111
I download th last version of Avs 2.6MT. Where do I put the file diff.7z ??? I paste Avisynth.dll infor SysWow64. But I have no clue what to do with the other file. Help me please !
matfra is offline   Reply With Quote
Old 9th September 2011, 18:42   #232  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
The diff file just contains the source code changes from the official Avisynth 2.6. Only developers need this file, you can ignore it.
-Vit- is offline   Reply With Quote
Old 11th September 2011, 22:36   #233  |  Link
SubJunk
Registered User
 
Join Date: Jun 2010
Posts: 441
Been getting occasional crashes with the new version
SubJunk is offline   Reply With Quote
Old 11th September 2011, 23:36   #234  |  Link
mbcd
Registered User
 
Join Date: Dec 2008
Location: Germany
Posts: 115
Yes, me too

2.5.8 runs like a charm
2.5.8 MT runs at 100% Processorusage (at all cores) but crashes with nearly (95%) every job.
2.6.0 MT the same as 2.5.8

Stability is a big problem ...
mbcd is offline   Reply With Quote
Old 12th September 2011, 00:00   #235  |  Link
SubJunk
Registered User
 
Join Date: Jun 2010
Posts: 441
Sorry, I should have been more specific with my previous post.
I meant the new version of 2.6 compared to the older 2.6; 2011.07.19 has the occasional crashes while 2009.09.19 works great for me.
SubJunk is offline   Reply With Quote
Old 12th September 2011, 21:50   #236  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Update, two more MT bugfixes.

I run into crash with colorspace conversion and so disabled USE_DYNAMIC_COMPILER. No crashes but slower. If speed of it is concern for you - post here, if there is enough demand I can investigate the issue. Solved, see below.

Last edited by SEt; 13th September 2011 at 01:04.
SEt is offline   Reply With Quote
Old 12th September 2011, 23:12   #237  |  Link
SubJunk
Registered User
 
Join Date: Jun 2010
Posts: 441
Thanks for the update, will test
SubJunk is offline   Reply With Quote
Old 12th September 2011, 23:33   #238  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@SEt,

In convert_planer.cpp, unpckbuf and packbuf are class local this is not thread safe.

Uncomment the _aligned_malloc and _aligned_free in the two GetFrame calls. Probably should test for malloc failures as well.
IanB is offline   Reply With Quote
Old 13th September 2011, 00:45   #239  |  Link
jeremy33
Registered User
 
Join Date: Jun 2009
Posts: 172
Thanks for the update Set.

All optimizations are good to take. That said I understand that it takes a long time and I prefer to have a stable soft rather than a faster soft unstable

jeremy33 is offline   Reply With Quote
Old 13th September 2011, 01:11   #240  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Another update, redownload if you just got the build earlier today.

IanB, it should not be a problem in MTMode 2, so I left it as is for now. Found the actual crash reason: I was playing with default calling convention and missed unspecified one on dynamically generated code call. I think conventions should be explicitly written where they are important.
SEt is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:05.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.