SEt's Avisynth 2.5.8 MT compiled for *X86_64*, Latest Build 4/16/2010 [Archive]

View Full Version : SEt's Avisynth 2.5.8 MT compiled for *X86_64*, Latest Build 4/16/2010

Pages : [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

JoshyD

16th February 2010, 10:53

Edit: Quicklinks Updated 4/26/2010

Featured Release 4/16/2010
+ Resize artifacting fixed
+ Horizontal resize code re-written to use SSE registers
+ Worth noting, often used functions Temporal Soften, Merge, etc have been tweaked for a decent speed gain
+ Bug found, fixed in memory copy routine, again
+ Universal binary, no longer need to distinguish between AMD and Intel builds
+ Optimized BitBlt memory copy Routine
+ Started implementation of SSE3/4 specific instructions when supported processor is detected
+ Removed most code paths intended to support CPU's without mmx/iSSE
+ Resize functions reworked to take advantage of extra registers available when processor is in 64bit mode
Avisynth64 binary and installer (http://avisynth64.googlecode.com/files/avisynth64_4-16-10.rar) built on 4/16/2010

Many of the plugins have been optimized and recompiled, please get the latest and greatest versions with this release.

Service update on 3/19/2010
+ Minor fixes to allow better usage of MT modes
+ Tweaks to code for small performance increases all around
+ Fixed resize bug
Use this build for Intel processors (http://avisynth64.googlecode.com/files/avisynth64_intel_3-19-10.rar)
Use this build for AMD processors (http://avisynth64.googlecode.com/files/avisynth64_amd_3-19-10.rar)

Version from 3/15/2010:
64 bit Avisynth 2.5.8 w/multithreading (http://www.mediafire.com/?mzh2zxmx1nw)

------------------------------------------------
Plugins (Alphabetically, for now)
------------------------------------------------

New on 3/21/2010
AddGrainC x64 (http://www.mediafire.com/?imj5nkwdkj1)

Built on 3/14/2010
AutoCrop x64 (http://www.mediafire.com/?mzddfmjjdyx)

New on 3/21/2010
aWarpSharp x64 (http://www.mediafire.com/?mjmfnuwf2ky)
This version is based on SEt's original rewrite found in this thread (http://forum.doom9.org/showthread.php?t=147285)

Built on 3/20/2010
Color Matrix x64 (http://www.mediafire.com/?ztjwmywqmm4)

New on 3/13/2010
DFTTest x64 (http://www.mediafire.com/?oemygnmkrji)-->needs the included libfftw3f-3.dll to be in your system32 directory

Built on 3/19/2010
DgDecode 1.5.8 x64 (http://www.mediafire.com/?c0wmemj5jam)
Note: Is missing some IDCT modes, will get them back ASAP

Built on 4/10/2010
EEDI2 x64 (http://www.mediafire.com/?znmyzdo2ize)
+ Vectorized main loop
+ Further restructured main loop to minimize branching, processor dependent speed increase

FeildHints x64 (http://www.mediafire.com/?ynkidzz4joz)

Built on 3/12/2010
FFT3DFilter (http://www.mediafire.com/?2ymcyfuzzzw)-->needs the included libfftw3f-3.dll to be in your system32 directory

Built on 3/15/2010
FFT3DGPU x64 (http://www.mediafire.com/?2chnt1jkwwm)
note:The hlsl (shader program) file is edited from the original to adhere to pixel shader 3.0 syntax rules. Please make sure to place the correct file in the same directory as the 64bit plugin.

New on 3/13/2010
kemuri-_9's FFMS2 (The Fabulous FM Source 2) (http://kemuri9.net/dev/avs/ffms2/x64/ffms2.dll)
Big thanks to kemuri-_9 for the build

Built on 3/29/2010
GradFun2DB x64 (http://www.mediafire.com/?w0trndmni3j)

Built on 4/08/2010
hqdn3d x64 (http://www.mediafire.com/?gyvmmzx0v4z)

Built on 3/14/2010
LeakKernelDeint x64 (http://www.mediafire.com/?tqmjdnjdnum)

Built on of 3/12/2010
MaskTools 2.0a41 (http://manao4.free.fr/masktools-v2.0a41.zip)
+ Now straight from the source, Manao

Built on 3/31/2010
MVTools2 x64 (http://www.mediafire.com/?2yjmze2ommz)
+ Continued conversion and updating of assembly functions
+ Removal of some code intended to support processors without mmx/iSSE
+ Converted often used assembly functions to SSE2 instead of mmx/iSSE
+ Updated to latest shared function library from x264
+ Healthy 20%+ speed increase over x86 version in most cases

New on 3/14/2010
TDeint x64 (http://www.mediafire.com/?kmcztm1xzjm)
This is basically the same as squid80's build, main differences being newer avisynth.h and newer compiler

TelecideHints x64 (http://www.mediafire.com/?wnemmzntgnh)

Built on 3/13/2010:
TIVTC x64 (http://www.mediafire.com/?i2qtli1mxik)

New on 3/20/2010
TNLMeans_x64 v1.0.3 (http://www.mediafire.com/?y4e3zd2zodd)

New on 3/20/2010
TTempSmooth x64 v0.9.4 (http://www.mediafire.com/?zv0jm3mtmzf)

RemoveGrain x64 (http://www.mediafire.com/?qjmyjt52miz)

Repair x64 (http://www.mediafire.com/?q00r32yegnh)

VerticalCleaner x64 (http://www.mediafire.com/?mjctqyjnoxh)

Visit Squid80's homepage for more x64 plugins (http://members.optusnet.com.au/squid_80/)

Benchmarking Suggestions

Here is a 64bit avs2avi for benchmarking (http://www.mediafire.com/?yykmkawa40j).
You can run it against the original (http://moitah.net/)

To simply run the script through Avisynth, execute the following at a command prompt:
avs2avi64.exe <path:\script.avs> -o n

A dialog box will pop up asking what to do for compression. Choose no recompression (or whatever similar option your os gives you) and the script will run without saving an output.

The same should be done with avs2avi.exe.

This will take two factors out of the speed equation: 64bit vs 32bit compressors and hard disk write speed. The final fps report from both runs will allow a fairly apples to apples comparison of the two builds.

Source Code
For those who are interested: The source is now hosted over at google code, I'll keep it as up to date as I can (http://code.google.com/p/avisynth64/)
The source is constantly in flux.

This wiki page (http://code.google.com/p/avisynth64/wiki/PluginLinks) has all of the plugins linked as well.

The source to any of the plugins I have personally modified is available upon request. Please message me if interested.

Chikuzen

16th February 2010, 15:30

Hi, JoshyD

I tried to test it with my Q9450 and 64bit windows7, but some DLLs seems to be insufficient.
where can i get libiomp5md.dll ?

JoshyD

16th February 2010, 20:06

Hmm, it appears that even when linking all your libs statically, intel's compiler still links the openmp libs dynamically, I'll get a rebuild up ASAP. From the ICC forums:

"The 11.0 Windows compilers (both C++ and Fortran) have decoupled /MT from having any effect on which OpenMP runtime (static or dynamic) gets linked. In fact, all of /MT[d], /MD[d], and /ML[d] (latter VS2003 only) now only effect which MS C runtime is linked.
We made this change because we want dynamic to be the default when linking the OpenMP RTL. The use of static OpenMP libraries is not recommended, because they might cause multiple libraries to be linked in an application. The condition is not supported and could lead to unpredictable results. It can also cause Thread Checker false positives and other problems with the Intel Threading tools.
If you want to link against the static OpenMP RTL, you must add /Qopenmp-link:static, which is a new switch for 11.0. So to produce a purely static executable, compile/link with /MT /Qopenmp-link:static
Patrick Kennedy
Intel Compiler Lab"

I didn't build the project with OpenMP libs enabled, but did allow the compiler to auto-parallelize loops it found it could. Perhaps this is the issue.

JoshyD

17th February 2010, 00:48

There is support for SetMTMode and it's functions, but not MT("command") as of now. The whole MT function is contained in an entirely separate DLL loaded form your plugins dir automatically, or manually at the beginning of the script.

The RAR does contain a copy of direct show source for use with AVI synth. Any program that can open avs files AND is already a 64 bit executable should do the trick. VDub64 comes to mind, as well as media player classic home cinema 64. 64 bit builds of x264 that specifically have the ability to open avs files should work as well.

I have personally tested Virtual Dub 64, it has played back all of my little test cases I've had a chance to run without any complaint.

For some 64bit plugins that MAY work with this DLL please check out Squid80's prior work. (http://members.optusnet.com.au/squid_80/) He was the guy who ported Avisynth 2.5.5 to x64 a while back, without the nicety of having a compiler that supports inline asm.

Other than that, the the top two plugins that I want to see work with 64bit are MvTools2 and FFT3DFilter. I get a lot of mileage out of those two projects.

If there's enough interest generated I'm considering going back and optimizing all the ASM routines to take advantage of architectural changes that have occured over the past 5 or so years. Some of these assembly routines are long in the tooth, and could be better tuned for newer processors (I think).

Anyhow thanks for the post, and keep checking back for updates.

Stephen R. Savage

17th February 2010, 00:54

JoshyD

17th February 2010, 01:33

What I meant was, could you modify your avisynth64 release to use a separate directory for autoloading 64-bit plugins to avoid name-conflicts and similar issues? On an aside, having mvtools2 + MaskTools2 would allow a lot of script functions to work automatically.

Sure, that's pretty easy. I can add throw that one on the to do list. I'm just a bit more focused on making sure others can run it at the moment.

Stephen R. Savage

17th February 2010, 03:50

JoshyD

17th February 2010, 05:02

Update: I copied avisynth.dll and devil.dll to system32 on a Windows 2008 R2 setup, after which I imported avisynth.reg. I created a test script with the code "BlankClip().ConvertToYV12()". It does not load in either VirtualDub64 or x264. Both crash upon exit. VirtualDub64 gives an error message "AVI Import Filter error: (Unknown) 80040154".

Edit: Processor is Core 2 T7250 "Merom".

This works for me, did you add the registry key that's included in the rar? When VDub throws that error, it usuaully means it can't find the right filter to decompress your stream.

I'm not sure which snapshot of the binary I compiled you may have grabbed, but along the way I realized I had the compile flags wrong. I was only generating code for a core I3/I5/I7. This caused my Penryn laptop to die when trying to load a clip. With the latest file up there, my Penryn executes no problem.

Just to be double safe, try this compilation of the avisynth binary:

It only has the SSE2 code path enabled (http://www.mediafire.com/?ykdjez2eomy)

If that doesn't work, we'll come up with a solution, I'm a bit hazy on how Windows associates registry keys with filter types. If anyone else has some pointers, that'd be great too.

Stephen R. Savage

17th February 2010, 05:16

The copy you just linked to depends on libiomp5md.dll, so I can't run it. Incidentally, my Merom CPU supports flags up to SSSE3, but not SSE4.1 like the higher-end Penryns. Perhaps you could use runtime CPU detection instead of requiring a specific instruction set compatibility?

Edit: Success! The build in the topic post of this now works. I guess it must have been silently updated.

Edit2: DirectShowSource and Spline36Resize work. However, scripts in my Avisynth32 plugins directory don't seem to be autoimported.

Edit3: TDeint64 built by Squid80 works fine. RemoveGrain64 by Kassandro does not (unknown exception).

Edit4: EEDI2_64 by Squid80 does not work. I believe this is because it is also statically linked to OpenMP, which according to a Google result on Intel's webpage, can cause conflicts.

Edit5: ConvertToRGB32 does not work. It always errors with the message "Rec.709 and PC Levels support require MMX and horizontal width a multiple of 4" regardless of settings/input.

JoshyD

17th February 2010, 06:10

EDIT:
You were right about EEDI2 using OpenMP and being dynamically linked to the libraries. I just recompiled the source with the OpenMP libraries statically linked:
EEDI2 64bit Multithreaded (http://www.mediafire.com/?mjomud5wnxd)

Also, where did you run across RemoveGrain64? I'd like a copy (and the source if possible) so I can figure out what exception it's throwing.

The only thing I can say for sure is that converting RGB error is hard coded in there for now, as there was a decent amount of assembly involved in converting the routine, and I just plain didn't feel like it. I'll get back around to it, put that on my todo as well.

I'll start checking into those other two plugins, I'm working on MVTools at the moment . . . that would be a huge win.

The other issue with the code paths was that I was basically telling ICC to target just my host CPU. When I tried to get it going on any other computer I realized my mistake. The current DLL for download has code paths for all EMT64 supporting Intel processors.

Once again, don't know if ICC cripples AMD chips or not . . .

Chikuzen

17th February 2010, 20:51

Hi, JoshyD
Thx for static build version. it seems to work at present without troubles.

I did some benchmarks. the results is here (http://en.pastebin.ca/1800425).

I think that faster 64bit decoder is necessary for me...

Stephen R. Savage

17th February 2010, 21:22

Kassandro posted his RemoveGrain64 on his own forum:

http://videoprocessing.11.forumer.com/viewtopic.php?p=700&sid=979b52936a673a92c52cd20f1c9425d0

Edit: Your rebuilt EEDI2() causes artifacts in chroma.
Edit2: Seems to be a pitch error. Doing a TurnLeft/TurnRight works around it.

JoshyD

18th February 2010, 21:35

Thanks for the benchmarcks! I don't think we're going to see any speed increases with 64bit code unless the assembly is re-worked to take advantage of more registers / less memory access.

Any resizers you run through there are really just using their old 32 bit counter part, essentially. The size of the pointers change, but the register usage at the CPU level is still the same, because it was specified explicitly. I'm going to continue work to see if I can't eek out some extra performance. I mainly want to see if I can't get some of the more demanding plugins a speed boost.

I'm not sure what's wrong with EEDI2, the source was straight compiled again.

Stephen R. Savage

19th February 2010, 01:05

JoshyD

19th February 2010, 12:06

Perhaps there was always the bug in the source code. Nevertheless, if you have experience in Avisynth plugin development, perhaps you could squash it for us? Please?

EEDI2 would normally not be anywhere near the top of my priorities, but without nnedi2, it's pretty much a necessary step to get TempGaussMC working on avs64 (the other steps being removegrain64, mvtools2_64, and masktools2_64).

Edit: EEDI2 bug squashed, I think
Try this out and let me know if it produces consistent results (http://www.mediafire.com/?oj2yhm0j2gn)

I DO have a somewhat working MVTools2. In so far as I have tested it, the "important" functions are working. A TON of the ASM has been re-coded to adhere to function calling specifications set forth by x64 c++. I did it by hand, meaning, there's probably a decent chance you'll crash it.

I also had to update the parts borrowed from other projects (xvid, x264, fftw), so those are a little "rough" at the moment.

My test cases mainly focused around motion vector generation and the degraining functions. Perhaps someone else will be able to fault it in other places, allowing me a chance to find and fix the bugs.

Here's the link to MVTools2 x64 (http://www.mediafire.com/?jf13dkynktj)

Personally, I see a significant performance increase (from ~20fps x86 to ~30fps x64, when using multi threading in both cases) when just writing out a raw stream. Try it out, and let me know where the problems are.

This is a little sample of what I've been using to mess around with the parameters. You can get it to go through a surprisingly large number of code paths just by varying the inputs to different degrain functions.

#MVTools x64
SetMTMode(2,4) #could be more, my system has four logical threads, but in certain instances more increase my encoding fps
LoadPlugin("D:\Development\mvtools2.dll")
AviSource("D:\testfile.avi")
ConvertToYV12(interlaced=true)

function MDegrain2i(clip source, int "overlap", int "dct", int "blksize", int "pel", int "search", int "searchparam")
{
overlap = default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct = default(dct,0) # use dct=1 for clip with light flicker
blksize = default(blksize, 8)
pel = default(pel, 2)
search = default(search, 4)
searchparam = default(searchparam, 3)

fields = source.SeparateFields() # separate by fields
super = fields.MSuper(chroma=true, pel=pel)

backward_vec2 = super.MAnalyse(blksize=blksize, isb = true, delta = 2, overlap=overlap, dct=dct, truemotion=true, temporal=true, pelsearch=pel, search=search, searchparam=searchparam)
forward_vec2 = super.MAnalyse(blksize=blksize, isb = false, delta = 2, overlap=overlap, dct=dct, truemotion=true, temporal=true, pelsearch=pel, search=search, searchparam=searchparam)

backward_vec4 = super.MAnalyse(blksize=blksize, isb = true, delta = 4, overlap=overlap, dct=dct, truemotion=true, temporal=true, pelsearch=pel, search=search, searchparam=searchparam)
forward_vec4 = super.MAnalyse(blksize=blksize, isb = false, delta = 4, overlap=overlap, dct=dct, truemotion=true, temporal=true, pelsearch=pel, search=search, searchparam=searchparam)

fields.MDegrain2(super, backward_vec2,forward_vec2,backward_vec4,forward_vec4, thSAD=500, thSCD1=500, thSCD2=130, plane=4)
Weave()
}

return MDegrain2i(last, overlap=4, blksize=8, pel=2)

Enjoy, and let me know any problems

aegisofrime

20th February 2010, 15:49

I do hope we can see TempGaussMC ported to 64 bit. That's one plugin that's slow but used a lot in the community. :)

Well, with MVTools and RemoveGrain done that's half the work over?

Stephen R. Savage

20th February 2010, 17:08

Fizick

21st February 2010, 19:17

JoshyD, you do big work!
I am intersted to see your TON of the ASM mvtools_64 source code lines ;)

tedkunich

22nd February 2010, 01:23

JoshyD, you do big work!
I am intersted to see your TON of the ASM mvtools_64 source code lines ;)

Fizick, you do know that he is Jeremy Duncan in another persona, right?

From the user control panel, Jeremy's last activity was on Feb 5. JoshyD joins on Feb 5... Hmmmmm

Stephen R. Savage

22nd February 2010, 02:35

I'm fairly certain that Jeremy Duncan would not be able to write any significant amount of code, much less port a large software project.

tedkunich

22nd February 2010, 06:19

I'm fairly certain that Jeremy Duncan would not be able to write any significant amount of code, much less port a large software project.

Does seem a stretch, but it is extremely suspicious that this guy comes here out of the blue at the same time our resident 64bit pest disappears...

JoshyD

22nd February 2010, 10:20

Does seem a stretch, but it is extremely suspicious that this guy comes here out of the blue at the same time our resident 64bit pest disappears...

I don't mean to be a pest about 64 bit binaries, I was just curious what improvements would be seen, if any. It was a nice chance to learn assembly, re-learn signal filtering, and all that good stuff.

Besides, Jeremy Duncan could barely compile the original 32bit source . . .

kemuri-_9

22nd February 2010, 14:06

Stephen R. Savage

22nd February 2010, 14:12

Actually, that is a good point. JoshyD, how about a 64-bit Avs 2.6? :D

Fizick

22nd February 2010, 20:06

Strange personality discussion.... I am simply waiting the source code and project files of MVTools2_x64 (GPL).
Are 32bit and 64 bit asm code generalized?

JoshyD

22nd February 2010, 20:20

my question this entire time of the thread is:
why are you patching a patched version of avisynth rather than the official version like the avs 2.6 cvs branch which has already had the multithreading features committed to it.

Well, I started with SEt's build because he had already gone through and shored up a lot of the problems with for loop variable scope. The more well behaved your code is to start, better it should compile.

It was my understanding that the 2.6 branch was still in alpha status. There's a fair amount of new functionality for new colorspaces that I didn't want to go messing with if they weren't done.

The 2.6 branch also makes more use of Softwire to dynamically generate assembly code. In the end, I honestly didn't want to deal with working around Softwire more than was necessary. Softwire is a bit long in the tooth to begin with, and the original developer took that project and it's now Swift Shader (http://www.transgaming.com/business/swiftshader/) with a somewhat different focus, and closed source.

I took what seemed "stable" at the time, and just started working around there. There's no reason I couldn't merge these changes into the 2.6 alpha, I just wanted to be sure I was starting from a stable base.

That being said, anyone care to clue me in on the status of 2.6 as of now? I don't want to step on anyone's toes, I just wanted to poke around the source in my spare time. If there's enough new/stable code in there, I could definitely focus some time/effort in getting that code to target x64.

It seems that the work being done going forward is focused on other areas more important than an x64 code base. This whole thing is a bit of an experiment / sideshow. I thought it was interesting enough to share, hence it's posting here.

I know the eventual goal is to get Avisynth working cross platform, as well. A targeted win64 release would also be useless in the long run, if that's where the project is going.

Side note/tangent:
The inference about cross platform support for Avisynth can really be seen by looking at the MaskTools code base. I'm pretty sure the documentation for it explicitly states that it was written with the intent of eventually being cross compiled. The assembly functions for it are ALL contained in ASM files, and there's a lot of them. The layout is really quite elegant, I like it quite a bit.

JoshyD

22nd February 2010, 21:09

Strange personality discussion.... I am simply waiting the source code and project files of MVTools2_x64 (GPL).
Are 32bit and 64 bit asm code generalized?

The source is finally up, sorry for the delay (http://www.mediafire.com/?yuzzwy3m1y1)

Unfortunately, they're not generalized as a whole. I did the inline assembler generally using just #ifdef's and then grabbed the latest x264 asm functions that the project uses, which are generalized. The actual asm for functions contained in files like bilinear.asm just got overwritten. It's trivial to swap the file for the original, and re-compile. It's just not exactly elegant.

The functions should have been generalized, but I was just working fast, kind of without putting a ton of thought into the process. I've been learning as I go, and it seems I always find an example of a sleeker solution after working through the first ugly one that popped into my head.

The main differences are in function calling, and register usage. You can get by with a lot less push/popping in 64bit land. Stack allocation between function calls changes as well. As a rule, all arguments are aligned at 8 byte boundaries. Arguments that are passed to the functions via registers still get shadow space on the stack, so your 5th integer argument will be at [rsp+40], if you didn't push any registers on the stack in the first place.

I wanted to ask some specific questions about the filtering and code copying functions. Was there a reason that they're often limited to the mmx registers? Things like the Horizontal Weiner filter are bothering me, because depending on your byte window, you're going to get different results from it, or it would seem that way. Right now, it has a 4 byte "window" to filter around. My understanding of a weiner filter is that it's adaptive, so changing its discrete window would change the filter's output altogether.

It's possible to look at 8 byte chunks using the XMM registers (unpacking to words for arithmetic (128bits total), repacking) but I'm unsure on the effect on overall image quality. Thoughts?

Finally, a lot of the mmx functions don't take advantage of the fact that we have a ton of XMM registers floating around that can also be used in mmx arithmetic. XMM0-XMM5 are all volatile across calls which could prevent some mmx registers from being shuffled around, etc.

When writing assembler, I'm not sure how the CPU's register files are architected to interact with each other. As in, if there's a pentalty associated with transferring a qword from an XMM register to an MMX reg, and vice versa.

I'm actually a VLSI designer (very large scale integration) (http://en.wikipedia.org/wiki/Very-large-scale_integration) by education, so thinking on the machine level is interesting and thought provoking. I don't know enough about the design of the x86 cores of late to generalize performance impact of various code paths. Is there any way other than running a battery of tests to analyze the clock cycles it takes for an instruction to retire?

I'm going to search around for the answers, but I thought I'd ask anyhow. Sometimes that's the fastest and most concise way to find the info you're after.

JoshyD

22nd February 2010, 21:40

RemoveGrain is not very useful without its brother Repair, but the version you linked works. Also, what version did you compile? The EEDI2 build works and is consistent, though no longer multithreaded. Will all filters with internal multithreading be incompatible with this version of Avisynth?

Edit: Strange request, but could you build TelecideHints (http://www.sendspace.com/file/gzvkzy)and FieldHint (http://www.sendspace.com/file/h08eea)as well? They're pretty useful for anyone who uses Yatta, even if Yatta itself isn't 64-bit, and should be completely free of assembly code.

Fixed:
Here is the repair function to try out. (http://www.mediafire.com/?q00r32yegnh)

I removed threading form EEDI2 because Avisynth can handle threads internally for its filters. When all these little binaries link to each other, there should be some arbiter of thread creation. At least that's what seems logical at the time of writing. If you tell Avisynth work with 4 threads, and then it takes one of those threads and instantiates a filter who goes off and spawns 4 more threads, it would seem that the situation would get messy after a bit. There would possibly be a lot of context switching going on, which would degrade performance overall.

Also, any filterd compiled to statically link to the OpenMP libraries cannot be instantiated alongside any other filter also statically linked to those libs, because this will cause openmp.lib to be initialized more than once. Apparently, this can degrade performance, at least according to the error that occurs upon crash. The plugins can be compiled to ignore this, but in the short term, I was trying to save people some trouble shooting head aches.

As for the other two plugins you requested, I'll see what I can do. If Avisynth 2.6 can withstand a 64 bit port somewhat seamlessly, I'm going to focus my attention there first.

Stephen R. Savage

22nd February 2010, 21:56

I get the error "LoadPlugin: unable to load 'Repair.dll'". Thank you for clarifying on the OpenMP issue. Again, I would like to ask what version of RemoveGrain/Repair your versions are based off. I ask this because there are two versions in "common" use.

Edit: ConvertToYUY2(interlaced=true) crashes your Avisynth64 build with SetMTMode(2) and causes an exception without SetMTMode.

Edit2: http://img43.imageshack.us/img43/50/avisynth64.png I'm liking the direction this project is going in. With masktools2 built, I can upgrade from "crude mvbob" (no error checking) to proper mvbob. Hopefully we can get some other volunteers to build plugins, so JoshyD doesn't have to do all the work.

JoshyD

23rd February 2010, 03:27

Edit: ConvertToYUY2(interlaced=true) crashes your Avisynth64 build with SetMTMode(2) and causes an exception without SetMTMode.

Which color mode are you coming from when the crash occurs? I can't seem to get mine to crash . . . I have actually re-written some of the color conversion since the original dll was posted.

Try this one instead (http://www.mediafire.com/?hnbymmnyxky)

If that's a no go, maybe a sample clip is in order?

You are quite correct about the repair I uploaded being completely incorrect. I'm not quite sure where that dll even came from, the final build is ~3x its size. As for the version, it says "1.0" at the top of the main source file, I started with the original attempted RemoveGrain64. There were some glaring oversights in it (not sign extending 32bit integers when using them to compute 64bit addresses) that needed to be corrected, but otherwise, everything looks quite similar to the version available from the author's website.

That being said, I think the post that RemoveGrain64 came from was made ~a week after the last version of RemoveGrain and Repair were released.

I also built the two plugins you asked about, they're linked in the first post.

tedkunich

23rd February 2010, 05:43

I don't mean to be a pest about 64 bit binaries, I was just curious what improvements would be seen, if any. It was a nice chance to learn assembly, re-learn signal filtering, and all that good stuff.

Besides, Jeremy Duncan could barely compile the original 32bit source . . .

My apologies sir... The subject matter and timing were just a too coincidental... :)

Stephen R. Savage

23rd February 2010, 05:43

You forgot to statically link OpenMP again. I was converting to YUY2 from interlaced YV12 (generated by SeparateFields().SelectEvery(4,0,3).Weave()) on a 640x480 format frame.

Edit: I googled for a copy of libiomp5md.dll and copied it to system32. I hope that's legal. Anyway, YUY2 conversion no longer crashes with the updated version. The new Repair version also works. Thanks for clarifying the version number involved. TelecideHints and FieldHint work as expected.

Edit2: For the record, Decomb64 by Squid80 works as expected (even with MT). TIVTC would be nice though.

Edit3: My remaining (major) wishlist, in descending order of priority:

mt_masktools
TIVTC
nnedi2 (yeah right)
GradFun2DB
dfttest

I suspect most of these will be a pain to port, possibly with the exception of GradFun2DB and mt_masktools. Getting a full stable avisynth.dll is of course top priority though.

Edit4: I believe the RemoveGrain/Repair version is the "pre-release" then. Confusingly, there is a "1.0 pre-release" and a "pre-release".

JoshyD

23rd February 2010, 06:51

OpenMP is open, so copying it to your system should be a ok, it's actually better than statically linking to the library, as it will allow the usage of multiple plugins that were compiled with parallel directives to be isntantiated concurrently.

Is there a major difference between the RemoveGrain and Repair versions?

Which version of mt_masktools are you looking for? I think 1.5.8 was the last build before 2.0 alphas started rolling out. I can port 1.5.8 very quickly, 2.0 will take a bit longer.

Stephen R. Savage

23rd February 2010, 07:33

patul

24th February 2010, 10:35

My apologies sir... The subject matter and timing were just a too coincidental... :)

Coincidental, (not to mention a word in *_iSSE.asm files :p ).

JoshyD

26th February 2010, 03:45

Latest v2.0 alpha (2.0a36). 1.5 stable isn't really used anymore. I think there was some rearrangement of the modes to Repair between the two versions, but I couldn't really tell you the difference to be honest. I think Didee mentioned something about it though.

Edit: You haven't answered this yet. How does the current build handle script/plugin autoloading? Is it simply disabled?

To be honest, I haven't even looked into the autoloading. The code is all there, in tact. It may be looking in Program Files instead of Program Files (x86) or something. I'll see if I can't get a straight answer on that. In the mean time, have fun with masktools 2.0a x64. (http://www.mediafire.com/?jc0zh3gcmwg) It's a very rough build, stress it out and let me know how it treats you.

Stephen R. Savage

26th February 2010, 04:06

mt_edge crashes when using the built-in kernels. A custom kernel seems to work fine.

TempGaussMC(edimode="EEDI2") also raises an exception, but I can't seem to trace it to a specific line.

JoshyD

26th February 2010, 04:20

Can you post a few of the scripts you're using to test it? I don't have a lot pre-written myself, nor a lot of good material to test with, help on either would be greatly appreciated.

Stephen R. Savage

26th February 2010, 04:44

DirectShowSource("file.avi")
mt_edge(param) # param = "sobert" or "roberts" or "laplace"

causes an instant crash.

I have traced the TGMC exception down to a line unrelated to masktools. It occurs due to TemporalSoften at the line "temporalsoften(2,255,255,28,2)".

The following mt_masktools-based scripts seem to work:

AntiAliasing (custom version)
DeHalo_alpha
FastLineDarken
LimitedSharpenFaster
SRestore (except double blend modes, depends on Average.dll)

This means that so far the following operations are already enabled in Avisynth64:

Motion-adaptive deinterlacing (TDeint, EEDI2)
Crude Inverse Telecine (TDeint + Decimate)
Deblending (mt_masktools)
Sharpening (mt_masktools)
Denoising (MVTools2)

JoshyD

27th February 2010, 03:26

That was embarrassing . . . it was a simple typo in the source on my part. Link is updated with version that should work correctly. Let me know of any more problems.

Stephen R. Savage

27th February 2010, 05:00

Thanks. mt_edge defaults work now, and every example usage of masktools I have seems to perform as expected. Please look into fixing the temporalsoften bug, as temporalsoften is (ab)used fairly often in scripts.

JoshyD

27th February 2010, 07:32

aegisofrime

27th February 2010, 11:34

Alright, I think I've got that temporal soften bug squashed, test it and let me know if there are any issues

I have TempGaussMC running . . . I'm showing some significant speed differences, anyone care to try there hand at it? I've got a four core machine, and telling AviSynth to use ~8 threads seems to be the sweet spot for processor utilization for me.

I want to try it! What are the files that's needed? I'm clueless about running 64-bit Avisynth.

JoshyD I take it you have a Core i7 machine? I wanna see what my Phenom II 955 gets.

Stephen R. Savage

27th February 2010, 16:37

JoshyD

27th February 2010, 18:46

I can confirm that TGMC/EEDI2 works now.

@aegis: You want avs64, eedi2_64, mvtools2_64, mt_masktools64, removegrain64, and repair64.

Things left to do:
1) Port changes to avs 2.6
2) Implement RGB code
3) Port more plugins
4) Clean up code

@aegis
Stephen's spot on about what to get, the directions for installation of the 64bit dll are pretty straight forward, and contained in the readme that accompanies the archive containing Avisynth64. Any headaches you encounter, come back and we'll (myself and whoever else cares to pitch in(I bet Stephen wouldn't mind)) will be as helpful as possible.

@Stephen
Your list of todo's is quite well put. I wanted to ask you if you saw any speed improvement under the 64bit environment. I went from encoding ~10fps on a 720x480 interlaced dv source to ~20fps. I feel like that's too big of a leap and I may have screwed something up along the line. If your numbers are anything similar, I can rest a little easier.

Code clean up is a big concern . . . it's kind of a mess in there, especially the hack and slash that was done to get the inline assembler in Avisynth to compile. I would really like to rip the assembly out to standalone ASM files and optimize it specifically for x64 to truly take advantage of the extra registers all around. I think there's some extra performance to eek out if that can be done. Any current inline ASM doesn't use the extra registers at our disposal. In addition, a lot of it uses the mmx register set, when we could easily use the SSE registers, which are twice as large and twice as numerous (8 64 bit mmx registers vs 16 128bit xmm (SSE) registers).

Some of the existing code can be directly dropped in to what's in Avisynth 2.6, while other portions of the project have gone to Softwire for dynamic assembly generation. That's just a pain to deal with, I would have rather seen assembly generated dynamically via macros and such as Masktools did. This would mean generating every function needed before runtime, leading to possibly larger code, but removes the dependence on Softwire, which is no longer officially maintained.

Seriously, I'm really impressed with the way the Masktools authors implemented their assembly routines. It obscures a lot of the function generation via a TON of macros, but with some architecture specific aliasing of registers/variables, the codebase can be compiled for both x86 and x64 without too much of a headache. That's almost the state I have it in now, an hour of coding and the source would compile for either architecture. It's also compiler agnostic (for the most part, there were some pieces of C++ that ICC and GCC don't agree with, which were coded around in my release-->most compilers like their function templates outside of *.cpp source files). If any Masktools authors care, I can give you the source I made, and point you to where the more strict compilers complain.

The RGB code should be back in the main DLL without too much work, I think I've almost got it taken care of, hopefully we'll see it back by the end of the week. It just wasn't on the top of my priority list, I wanted to get TempGaussMC up and going before anything else.

The script produces awesome results for restoring my old home movies, but was painfully slow when comparing it to a quick noise removal / deinterlace with some of the other plugins (namely a quick run through FFT3DFilter and MVDegrain coupled with yadifmod/nnedi2). The x64 results appear visually identical, so I don't think there are any steps accidentally skipped in the x64 code path. I'm racking my brain for differences outside of arhcitecture, but can't think of any as of yet.

That being said, working with the same samples, I can literally encode almost twice as fast when working under a pure 64bit environment. This makes re-encoding 24 hours of raw dv video (thank you HDD's in the TB+ range) seem like a practical batch job.

I've grabbed the 2.6 branch from the CVS periodically, and it's not stable for me in x86 land as of yet. The code compiles, and starts to serve the video stream, but output will just drop to zero a few hundred frames into an AVI encode. When the project matures to a point that the authors feel comfortable calling it a beta release, I want to go to work. A lot of the code can just be "dropped in" while I'd like to create a more elegant solution to replace Softwire generated assembly. This would make the whole package a bit more future proof. We're going to see x64 stick around for a while, it makes sense to start to use the features it offers. I personally have 4 x64 machines floating around, while clinging to an old x86 P4 (northwood 2.6ghz) that I use to mess around with various linux distros.

Final note: Both GCC and ICC support iterative compile-->runtime profiling of code-->profile based optimization. None of that has been done for any of these releases. When it's all a bit more stable, I'd like to allow some time for this. The guys at doom10 recommend this approach when doing your own x264 compiles. (http://doom10.org/index.php?topic=26.0)

kemuri-_9

27th February 2010, 19:36

Final note: Both GCC and ICC support iterative compile-->runtime profiling of code-->profile based optimization. None of that has been done for any of these releases. When it's all a bit more stable, I'd like to allow some time for this. The guys at doom10 recommend this approach when doing your own x264 compiles. (http://doom10.org/index.php?topic=26.0)

this seems like you're trying to imply that MSVC does not have a profiling and optimization feature like ICL/ICC/GCC,
which is untrue as MSVC does have PGO (Profile Guided Optimization)...
i only recall the feature only being in the professional/team versions and not the express/free ones though.

x264 is easier to profile as it's an executable, so you only need to provide a file to work with and it can profile.

avisynth profiling would be more complex as you need to supply both a script and a program to profile avisynth with...
not to mention going through all the code paths would also be a pain.

Stephen R. Savage

27th February 2010, 19:37

I think a key problem is a lack of ways to benchmark AVS64. I use avs2avi for standard AVS, but there is no analogue for AVS64. For example, if you benchmark against x264, the results are skewed by the 64-bit advantage for x264. If you normalize by using avs2yuv for the 32-bit Avisynth, you still have to factor in the speedup by cutting out the piping overhead.

That said, here are some benchmarks. All results were taken using x264 64-bit in q=51, preset=ultrafast. 32-bit AVS was fed using avs2yuv, which should have negligible performance cost when single-threading.

Edit: Performance benchmarks have been redone using avs2avi.

TempGaussMC/EEDI2
32-bit: 2.83 fps
64-bit: 3.01 fps

MDeGrain3
32-bit: 6.20 fps
64-bit: 6.90 fps

AAA (mt_masktools, EEDI2)
32-bit: 1.92 fps
64-bit: 0.63 fps (SetMTMode: 1.94 fps)

Didee's Edge Mask
32-bit: 80.16 fps
64-bit: 89.02 fps

EEDI2 Resize2x
32-bit: 5.59 fps
64-bit: 5.67 fps

None of these cases came out bit-exact. I am a bit confused as to why AAA() comes out so much slower when all the components are faster.

Edit: I have traced the AAA slowdown to the following code fragment:

input = DirectShowSource("640x480p30.xvid.avi")

ox = width(input)
oy = height(input)

aa = TurnRight(input).EEDI2(field=1).TurnLeft().EEDI2(field=1)

edge = mt_logic(mt_edge(aa, "5 10 5 0 0 0 -5 -10 -5 4", 0, 255, 0, 255),
\ mt_edge(aa, "5 0 -5 10 0 -10 5 0 -5 4", 0, 255, 0, 255), "max").Greyscale().
\ Levels(0, 0.8, 128, 0, 255, false).Spline36Resize(ox, oy, -0.5, -0.5, 2 * ox, 2 * oy)
ds = Spline36Resize(aa, ox, oy, -0.5, -0.5, 2 * ox, 2 * oy)
maskmerge = mt_merge(input, ds, edge, U=1, V=1)
MergeChroma(ds)

I think it is a cache-related bug, because none of the individual pieces is slower.

JoshyD

27th February 2010, 21:47

this seems like you're trying to imply that MSVC does not have a profiling and optimization feature like ICL/ICC/GCC,
which is untrue as MSVC does have PGO (Profile Guided Optimization)...
i only recall the feature only being in the professional/team versions and not the express/free ones though.

x264 is easier to profile as it's an executable, so you only need to provide a file to work with and it can profile.

avisynth profiling would be more complex as you need to supply both a script and a program to profile avisynth with...
not to mention going through all the code paths would also be a pain.

Oh, I know that MSVC does profile, but GCC and ICC are just "better" compilers. If anything, at least give GCC credit for being free, cross platform, having auto parallelization and auto vectorization, and always being up to date. That's a lot for a community driven project. It constantly produces code similar in quality to the much more expensive and commercially developed ICC, and VS2008's compiler is pretty dated, I believe. It will still compile some really questionable coding practices without as much as a warning. I haven't looked at the release candidate for 2010 yet though.

Note:
Compilers are stupid, I've written them in the past. Consider them akin to something like google's translator. The more straightforward you are with the language, the easier it is for a compiler to understand. Hence, I don't like VS2008's lax rules for C++ syntax. Assembly is pretty much as straight forward as it comes, but inlining it into C++ functions is generally a bad idea. You tell the compiler "make me some machine code from this high level language" and you give it machine code wrapped in high level language? Why translate the same sentence twice??

And that ends my rant . . . sorry, back to DLL optimization.

A dll is essentially an executable as well, it just can be dynamically tagged on to whatever program needs it at the time. It contains a set of functions that are then made available to the process that utilizes it. As avisynth is a frame server, it has one purpose, to provide video and audio to the calling program. The entry point into avisynth's dll and subsequent function calls internally shouldn't depend on the program.

The drawback to dll profiling is that you can only profile functions in Avisynth that are called internally by Avisynth. The compiler doesn't know what the external program is doing to the registers, memory, etc, so no profile can be created for any externally called functions. Things like internal filters will behave the same each time they are called, because external programs cannot call something like a resize function directly. It is my understanding that useful profiles can be generated for these. The machine state is going to be known prior to each internal function call.

It would DEFINITELY be painful to try and get every code path to execute on avisynth, while also also providing a number of test videos to stress them.

Just feeding working scripts with strings of useful functions (pseudo-random perl avs generation anyone?) into x264 repeatedly at the command line would hit a lot of the code paths. Random test vector generation has been useful for me in the past, it may fail miserably here, but if I get the time I may try it out. Hey, it's my own time to waste, I'm not suggesting anyone even think about it.

JoshyD

27th February 2010, 22:00

I think it is a cache-related bug, because none of the individual pieces is slower.

Time to go bug hunting!

Just converted and restored RGB functionality, it's up and in the latest release on the front page.

kemuri-_9

27th February 2010, 22:04