SEt's Avisynth 2.5.8 MT compiled for *X86_64*, Latest Build 4/16/2010 - Page 9

osgZach · 14th March 2010, 20:58

Quote:

Originally Posted by JoshyD

@noee

@Squid80
You've got a DGDecode listed on your webpage along with source, but checking the version info indicates it's 1.4.6. Any chance you have the 1.5.6 source on hand and I could take a peek at it? Also, may I add that to the first post?

DGIndex/Decode/VFAPI sources are all available at Neuron2's web site. Which was why I was surprised there were no newer x64 compiles..

http://neuron2.net/dgmpgdec/dgmpgdec.html

Also some other things we may not have can probably be found here.

http://www.neuron2.net/mine.html

levi · 14th March 2010, 21:34

Quote:

@levi
Why only 3 threads? Are you accounting for x264 also taking up some cores? I think x264's defaults to creating 1.5x the amount of threads it detects your system can run. I was hoping for some more speed gains. Any chance on seeing some straight single threading tests on the same system? I want to re-build tdeint to see if I can't eek out some more performance as well.

3 MT threads plus letting x264 do what it wants usually gives me the best performance.

I tested single threaded as you requested - it gives same 7-8% increase.

tdeint is pretty slow. As was mentioned, leakkerneldeint is a much faster one. I would much rather see you give that one a go vs. trying to get further performance gains with tdeint.

@Squid80
thanks for dgdecode 1.5.4 x64

noee · 14th March 2010, 21:41

Quote:

I'm guessing this means that you've gotten tivtc with working results? If so, that's great news.

Yes, I've reviewed the results and it appears to be correct. I'll run a couple of more tests a little later today.

kemuri-_9 · 14th March 2010, 21:50

Quote:

Originally Posted by JoshyD

@kemuri-_9
Can I link your FFMS2.dll on the first post? I can move it over to mediafire if you don't want to waste bandwidth on hosting it locally.

sure, you can post it in the first post and leave it how it is on referencing my website (i'm totally not concerned about the bandwidth for the server it's on)
I'll generally be updating it as bug reports come in, so it would be helpful for me to update it directly.

JoshyD · 14th March 2010, 22:50

Quote:

Originally Posted by levi

tdeint is pretty slow. As was mentioned, leakkerneldeint is a much faster one. I would much rather see you give that one a go vs. trying to get further performance gains with tdeint.

Did a quick and dirty port of LeakKernelDeint, link is on the main page. Nice to have it for use in avisynth64, but it's hardly speed limited to begin with. This version should be at least as fast as the 32bit plugin.

levi · 14th March 2010, 23:51

Another note, this build has the same flaw as Set's previous builds - If I turn hyper-threading off in the bios (therefore only using my 4 xeon cores) the same test (with your 64bit build) gives 31 fps - which is another 5% faster & completely counter-intuitive. It's a mystery that was never solved.

----------------------------------------------------------

much better performance with LeakKernelDeint port.

Xeon quad core E5530 2.40 ghz w/ no hyperthreading

x264(x86) w/ 5 threads + set avisynth 2.6(x86)
First Pass Output to null = 57.4

x264(x64) w/ 5 threads + JoshyD avisynth 3-13-10(x64)
First Pass Output to null = 63.5

I've seen an 10% speed improvement

Code:

SetMTmode(3,2)
mpeg2source("my.d2v")
SetMTmode(2,2)
ord = last.getparity() ? 1 : 0
LeakKernelDeint(ord,sharp=true)
crop(4,4,1916,1076)

LanczosResize(1280,720)

JoshyD · 15th March 2010, 02:55

Alright, I've gotten a DgDecode 1.5.8 built and running correctly on my system. It will only decode with SSE2 or SKALSSE algorithms for now, I'll work on getting the others back. Link is on the main page. It's not exactly x64 optimized, but at least it's the latest build.

squid_80 · 15th March 2010, 04:06

I'm not sure where the source code for my 1.5.4 build is. I didn't even know I had it.

The problem with DGDecode (the reason why I didn't compile new versions for x64) is that to get maximum speed, I found I had to hand compile some source files with ICC and others with VS - that's why I said it's possible the 1.5.4 build is slower than the older 1.4.8. Run some benchmarks if you want to check the difference. Also the source code only comes as a .zip file, there is no CVS/SVN/Git that allows for easy merging of files so updating it is tiresome.

levi · 15th March 2010, 05:29

FYI - CelticDruid has some 64 bit - multithreaded xvid builds @ http://esby.free.fr/CelticDruid/mirror/XviD/

1.2.0 CVS

1.3.0 CVS

MuLTiTaSK · 15th March 2010, 05:54

http://www.xvidvideo.ru/xvid-video-codec/

levi · 15th March 2010, 06:23

Quote:

Originally Posted by MuLTiTaSK

http://www.xvidvideo.ru/xvid-video-codec/

Sweet,

JoshyD

Think I found a bug...

LanczosResize(576,320) causes crash on Veedub64 open. LanczosResize(1280,720) works fine

aegisofrime · 15th March 2010, 11:01

Thanks for the DGDecode JoshyD! Now all we need is nnedi2 and my DVD deinterlacing toolchain is complete...

JoshyD: I finally got around to testing it on my Phenom II rig. Upon starting the script with x264 I get this error:

Here's my script if it matters:

Code:

loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\DirectShowSource.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\EEDI2.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\MT.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\mt_masktools25.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\mvtools2.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\RemoveGrain.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\Repair.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\FFT3dGPU.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\DGDecode.dll")
loadplugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\VerticalCleaner.dll")
import("C:\Program Files (x86)\AviSynth 2.5\plugins\TempGaussMC_beta2.avsi")
import("C:\Program Files (x86)\AviSynth 2.5\plugins\LimitedSharpenFaster.avs")

SetMTMode(2,4)
DGDecode_mpeg2source("D:\Raws\Test.d2v")
/*TempGaussMC_Alpha3(2, 1, 1, EdiMode="EEDI2")
crop( 0, 0, 0, -2)
Spline36Resize(720,480)

Distributor()

MT("LSFMod()")*/

As you can see, I have commented out everything but the source filter, to isolate the source of the error. It still persists with this.

osgZach · 15th March 2010, 16:20

Quote:

Originally Posted by squid_80

I'm not sure where the source code for my 1.5.4 build is. I didn't even know I had it.

The problem with DGDecode (the reason why I didn't compile new versions for x64) is that to get maximum speed, I found I had to hand compile some source files with ICC and others with VS - that's why I said it's possible the 1.5.4 build is slower than the older 1.4.8. Run some benchmarks if you want to check the difference. Also the source code only comes as a .zip file, there is no CVS/SVN/Git that allows for easy merging of files so updating it is tiresome.

I definitely noticed a big difference in speed after switching to the newer one.. I haven't tried the 1.5.8 compile yet (because I use reference for iDCT) but Squid's original older version is a hell of a lot faster..

The same script actually runs SLOWER than single threaded 1.5.8 in 32 bit.. I thought I broke something on my system when I started seeing the clock go up to 4... 5 hours. But I had to go to bed so I just let it run and when I woke up later and checked on it, it had frozen after hitting the 6,000 frame range.

Which was weird, the program was NOT hard locked or anything, the status window was just frozen.. nothing updating, etc.. But abord worked and the program was responding, etc..

Kind of makes me feel bad for bugging you guys for a more updated build now that I see its going to take a lot of manual hands on work to get it running decently. That plus I have to throw out 37 d2v's from 1.5.8 to go down to 1.4.6 based build, then had to throw those out to go to 1.5.6 or w/e the newer one he did was. heh.. Of course the real time-killer is redoing the metrics off of them in Yatta.. that takes a good 10 minutes or so for each one I think.. adds up to a few hours or something like that.

Looking at the readme for the one hosted on Squid's page it seems like it only does iDCT through SSE2 also ? It didn't complain so I figured all the modes were there, but was just curious HOW important a difference there is between the various iDCT modes.. I always use references cause I figured why not, its fast enough and is supposed to be the most "accurate".. But I really don't know what iDCT is about either, so.. just wondering.

Update
I've been giving the 1.5.8 compile a try and it looks pretty solid from what I can tell. It seems to have about the same performance as the previous 1.4.6 based compile by Squid80. The ETA is pretty stable below the 3 hour mark (goes up and down, currently 2h:35m) and I used SSE2 32-bit on the iDCT. This is the same script I posted back as working / non crashing under the 1.4.6 based version, so hopefully this one won't crash either. Averaging about 3.5fps according to rough math estimates ( (8500 frames / 40m) / 60s ) FPS meter runs anywhere between 2 to 5 fps.

Very good performance I think.

JoshyD · 15th March 2010, 18:47

@levi
I think there you did find a bug, I've done a quick fix for it, one I should have done a while ago. Can you try the latest DLL from the first post and see if that still crashes out? If so, some clip info may be in order so I can fix it properly.

@aegisofrime
One or more of those plugins was compiled with open mp enabled, and then statically linked. The error basically means more than one binary in your chain tried to initialize the library, which is discouraged. I don't think x264 uses OpenMP, I know my avisynth does in places, but the latest plugins shouldn't. Hence, there shouldn't be any errors. I've been known to be a bit inconsistent on my compile environments, but I thought I killed off open mp support in all the latest builds. I'll double check this, but could you get the latest build of each plugin as well to see if it still errors?

The option is out there to require the OpenMP dll be in your system32 directory, and then not statically link to it when building. I just haven't done this to reduce the headaches of installing another version of avisynth.

I'm guessing it'll throw errors on the resizing still, but that remains to be seen. I'm still working on an SSE3 only version of vertical resize function.

@osgZach
What's the final speed verdict on the 1.5.8 compile? Is it slower than the 32bit build? If that's the case, bummer. Squid80 mentions hand compiling the sources with different compilers, and that sounds gross. I'd like to avoid this, but I want DgDecode working as fast or faster in the 64bit environment.

To everyone and anyone:
A general question to those in the know about OSS, is everything I'm doing legal under the GPLv2? I've been kicking around the idea of hosting the source code somewhere (SourceForge, Google Code) and wanted to make sure this was alright with a) the community at large and b) the license under which Avisynth was released under.

I've also been wanting to set up a (poorly done) website somewhere for documentation and filter linking that doesn't rely on the main page of this thread. Additionally, I'd like to create a "swiss army knife" archive of the most useful plugins for single click downloading. Also, I think a real installer would be useful to many people, especially those hesitant that 64bit Avisynth will break 32bit Avisynth functionality.

Feedback, as always, is much appreciated.

Stephen R. Savage · 15th March 2010, 20:23

You are technically in violation of GPL, since you do not host the modified AMD64-compatible source code (for the plugins) or provide a written offer for it.

osgZach · 15th March 2010, 20:41

I haven't done exclusive testing of 32 vs 64-bit. However my guesstimation is that currently it should be at least as fast or very close.

I only started seeing these 2-1/2 hour encode times once I began using MT anyway. Under x32 it was 4h:45m and looked to be around 3-1/2 or 4h under x64 single threaded.

I know that's not a kosher analysis but you'll just have to trust me that at the very least, I find 1.5.8_x64 to be well within acceptable performance. It pretty much matches previous 1.4.6_x64 times so I think that's as good an indicator as any.

As for GPL issues.. Avisynths source code is freely available right? I think other modded editions have been let loose here too as well. As long as you provide the source and credits, release under GPL with your own updates, I don't think there should be an issue? Just got to get it taken care of. Currently I don't think anyone cares that much about technicalities, as you seem to have good intentions here. Just got to make sure future releases in an "official" type way that you follow the GPL rules.

I'm sure experienced members like the various utility authors here can help you understand any GPL questions and make sure everything is in order.

If you need hosting, I may be willing to donate some space and a pretty web page for you, and if the bandwidth is only a couple GB a month that should be fine.. If bandwidth becomes an issue you could just make a project on Sourceforge and keep everything distributed under there? Link to it from the pretty web page

JoshyD · 15th March 2010, 21:11

@Stephen
A written offer on the downloads page will suffice in lieu of packing up all the sources immediately and getting them hosted somewhere? Packing the sources for ~15 plugins isn't hard either, it's just a matter of linking them all somewhere, preferably not muddying up the first post.
Side note: some of the source is just plain embarrassing to have others look over. When I started porting, I couldn't write assembly AT ALL. I understood most of it, but comparing my earlier coding "style" (style=non-existant) to the macros and such I have now, there's significant difference. The more recent code at least looks halfway competent.

@osgZach
I may have to take you up on the offer of a pretty webpage! My mediafire download stats have the binary being downloaded just north of 500 times since I posted the original about a month ago. Granted, those are not all unique downloads, but at least there's been some interest in the port. I wonder if interest will continue? I don't see anything preventing me from merging x64 changes into 2.6 once it sees the light of day, either. At any rate, I'd like to generate better documentation, an installer, you know, kind of round out the port a bit. One step at a time, for now, GPL compliance, because, hey, open source software is pretty awesome.

osgZach · 15th March 2010, 21:25

Cool.. I will try to find some time within the next day or two, to do a fast and dirty mockup for a prospective site for you.

500 downloads isn't much as it is.. I don't think I've ever broken 50gb of usage in a single month, with my magazine and other randomly downloaded stuff I host sometimes. So I don't anticipate a huge problem.

squid_80 · 16th March 2010, 04:36

Back when I started avisynth64, Wilbert and IanB set up a branch for me to use in avisynth's CVS on sourceforge. Unfortunately I never got very far for various reasons (sorry!). But maybe they would let you take it over if you asked them.

JoshyD · 16th March 2010, 05:19

Could anyone with an AMD processor test this version of avisynth64 and let me know if the vertical resize functionality is restored?

14th March 2010, 23:51	#166 \| Link
levi Registered User Join Date: Mar 2003 Posts: 116	Another note, this build has the same flaw as Set's previous builds - If I turn hyper-threading off in the bios (therefore only using my 4 xeon cores) the same test (with your 64bit build) gives 31 fps - which is another 5% faster & completely counter-intuitive. It's a mystery that was never solved. ---------------------------------------------------------- much better performance with LeakKernelDeint port. Xeon quad core E5530 2.40 ghz w/ no hyperthreading x264(x86) w/ 5 threads + set avisynth 2.6(x86) First Pass Output to null = 57.4 x264(x64) w/ 5 threads + JoshyD avisynth 3-13-10(x64) First Pass Output to null = 63.5 I've seen an 10% speed improvement Code: SetMTmode(3,2) mpeg2source("my.d2v") SetMTmode(2,2) ord = last.getparity() ? 1 : 0 LeakKernelDeint(ord,sharp=true) crop(4,4,1916,1076) LanczosResize(1280,720) Last edited by levi; 15th March 2010 at 00:19.

15th March 2010, 05:54	#170 \| Link
MuLTiTaSK Registered User Join Date: Nov 2006 Posts: 668	http://www.xvidvideo.ru/xvid-video-codec/ __________________ StaxRip at Doom9's \| StaxRip at Sourceforge [RSS] \| StaxRip Donations \| TheMTiNFo

15th March 2010, 18:47	#174 \| Link
JoshyD Registered User Join Date: Feb 2010 Posts: 84	@levi I think there you did find a bug, I've done a quick fix for it, one I should have done a while ago. Can you try the latest DLL from the first post and see if that still crashes out? If so, some clip info may be in order so I can fix it properly. @aegisofrime One or more of those plugins was compiled with open mp enabled, and then statically linked. The error basically means more than one binary in your chain tried to initialize the library, which is discouraged. I don't think x264 uses OpenMP, I know my avisynth does in places, but the latest plugins shouldn't. Hence, there shouldn't be any errors. I've been known to be a bit inconsistent on my compile environments, but I thought I killed off open mp support in all the latest builds. I'll double check this, but could you get the latest build of each plugin as well to see if it still errors? The option is out there to require the OpenMP dll be in your system32 directory, and then not statically link to it when building. I just haven't done this to reduce the headaches of installing another version of avisynth. I'm guessing it'll throw errors on the resizing still, but that remains to be seen. I'm still working on an SSE3 only version of vertical resize function. @osgZach What's the final speed verdict on the 1.5.8 compile? Is it slower than the 32bit build? If that's the case, bummer. Squid80 mentions hand compiling the sources with different compilers, and that sounds gross. I'd like to avoid this, but I want DgDecode working as fast or faster in the 64bit environment. To everyone and anyone: A general question to those in the know about OSS, is everything I'm doing legal under the GPLv2? I've been kicking around the idea of hosting the source code somewhere (SourceForge, Google Code) and wanted to make sure this was alright with a) the community at large and b) the license under which Avisynth was released under. I've also been wanting to set up a (poorly done) website somewhere for documentation and filter linking that doesn't rely on the main page of this thread. Additionally, I'd like to create a "swiss army knife" archive of the most useful plugins for single click downloading. Also, I think a real installer would be useful to many people, especially those hesitant that 64bit Avisynth will break 32bit Avisynth functionality. Feedback, as always, is much appreciated. Last edited by JoshyD; 15th March 2010 at 19:41.

15th March 2010, 20:41	#176 \| Link
osgZach Registered User Join Date: Feb 2009 Location: USA Posts: 676	I haven't done exclusive testing of 32 vs 64-bit. However my guesstimation is that currently it should be at least as fast or very close. I only started seeing these 2-1/2 hour encode times once I began using MT anyway. Under x32 it was 4h:45m and looked to be around 3-1/2 or 4h under x64 single threaded. I know that's not a kosher analysis but you'll just have to trust me that at the very least, I find 1.5.8_x64 to be well within acceptable performance. It pretty much matches previous 1.4.6_x64 times so I think that's as good an indicator as any. As for GPL issues.. Avisynths source code is freely available right? I think other modded editions have been let loose here too as well. As long as you provide the source and credits, release under GPL with your own updates, I don't think there should be an issue? Just got to get it taken care of. Currently I don't think anyone cares that much about technicalities, as you seem to have good intentions here. Just got to make sure future releases in an "official" type way that you follow the GPL rules. I'm sure experienced members like the various utility authors here can help you understand any GPL questions and make sure everything is in order. If you need hosting, I may be willing to donate some space and a pretty web page for you, and if the bandwidth is only a couple GB a month that should be fine.. If bandwidth becomes an issue you could just make a project on Sourceforge and keep everything distributed under there? Link to it from the pretty web page Last edited by osgZach; 15th March 2010 at 20:45.

16th March 2010, 05:19	#180 \| Link
JoshyD Registered User Join Date: Feb 2010 Posts: 84	Could anyone with an AMD processor test this version of avisynth64 and let me know if the vertical resize functionality is restored? Last edited by JoshyD; 16th March 2010 at 18:29.

15th March 2010, 02:55	#167 \| Link
JoshyD Registered User Join Date: Feb 2010 Posts: 84	Alright, I've gotten a DgDecode 1.5.8 built and running correctly on my system. It will only decode with SSE2 or SKALSSE algorithms for now, I'll work on getting the others back. Link is on the main page. It's not exactly x64 optimized, but at least it's the latest build.

15th March 2010, 04:06	#168 \| Link
squid_80 Registered User Join Date: Dec 2004 Location: Melbourne, AU Posts: 1,963	I'm not sure where the source code for my 1.5.4 build is. I didn't even know I had it. The problem with DGDecode (the reason why I didn't compile new versions for x64) is that to get maximum speed, I found I had to hand compile some source files with ICC and others with VS - that's why I said it's possible the 1.5.4 build is slower than the older 1.4.8. Run some benchmarks if you want to check the difference. Also the source code only comes as a .zip file, there is no CVS/SVN/Git that allows for easy merging of files so updating it is tiresome.

15th March 2010, 05:29	#169 \| Link
levi Registered User Join Date: Mar 2003 Posts: 116	FYI - CelticDruid has some 64 bit - multithreaded xvid builds @ http://esby.free.fr/CelticDruid/mirror/XviD/ 1.2.0 CVS 1.3.0 CVS

15th March 2010, 20:23	#175 \| Link
Stephen R. Savage Registered User Join Date: Nov 2009 Posts: 327	You are technically in violation of GPL, since you do not host the modified AMD64-compatible source code (for the plugins) or provide a written offer for it.

15th March 2010, 21:11	#177 \| Link
JoshyD Registered User Join Date: Feb 2010 Posts: 84	@Stephen A written offer on the downloads page will suffice in lieu of packing up all the sources immediately and getting them hosted somewhere? Packing the sources for ~15 plugins isn't hard either, it's just a matter of linking them all somewhere, preferably not muddying up the first post. Side note: some of the source is just plain embarrassing to have others look over. When I started porting, I couldn't write assembly AT ALL. I understood most of it, but comparing my earlier coding "style" (style=non-existant) to the macros and such I have now, there's significant difference. The more recent code at least looks halfway competent. @osgZach I may have to take you up on the offer of a pretty webpage! My mediafire download stats have the binary being downloaded just north of 500 times since I posted the original about a month ago. Granted, those are not all unique downloads, but at least there's been some interest in the port. I wonder if interest will continue? I don't see anything preventing me from merging x64 changes into 2.6 once it sees the light of day, either. At any rate, I'd like to generate better documentation, an installer, you know, kind of round out the port a bit. One step at a time, for now, GPL compliance, because, hey, open source software is pretty awesome.

15th March 2010, 21:25	#178 \| Link
osgZach Registered User Join Date: Feb 2009 Location: USA Posts: 676	Cool.. I will try to find some time within the next day or two, to do a fast and dirty mockup for a prospective site for you. 500 downloads isn't much as it is.. I don't think I've ever broken 50gb of usage in a single month, with my magazine and other randomly downloaded stuff I host sometimes. So I don't anticipate a huge problem.

16th March 2010, 04:36	#179 \| Link
squid_80 Registered User Join Date: Dec 2004 Location: Melbourne, AU Posts: 1,963	Back when I started avisynth64, Wilbert and IanB set up a branch for me to use in avisynth's CVS on sourceforge. Unfortunately I never got very far for various reasons (sorry!). But maybe they would let you take it over if you asked them.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode