SEt's Avisynth 2.5.8 MT compiled for *X86_64*, Latest Build 4/16/2010 [Archive] - Page 2

JoshyD

28th February 2010, 00:46

@kemuri-_9

I like the way that x264 has done their asm coding, a lot. MvTools2 uses some portions of x264, and the fact that they have a nice and easy abstraction layer made porting those parts of the code seamless.

@Stephen
The issue with the AAA script you wrote is DEFINITELY cache related. Try SetMTMode(1,1) and it magically goes the speed it should when single threaded. It's ever so slightly faster in x64 mode, when you add threading to the mix, there are definite speed gains to be had. I have no idea why the normal caching mechanism is fussy, I didn't re-write any of that code.

I've been using VDub for benchmarking, but can do a quick recompile of those tools in a bit. I just write out raw streams taking any compressor out of the mix.

As for the subtle pixel differences, they're quite odd, and I can't explain them as of yet.

If comparing with ImageMagick (http://www.imagemagick.org/script/index.php), the average error of 24 bit bitmap captures for each channel is ~0.005%. I'm going to see if I can find what's causing the difference, there may be some rounding differences when working with 64bit integers that are playing a role. I'm going to fine tooth comb the code and see what I can see.

Edit: I just remembered that in the normal version of avisynth goes through Softwire for (I think) any planar to non-planar conversion, it very well could be that's where the difference lies.

Stephen R. Savage

28th February 2010, 01:26

I can confirm that SetMTMode(1, 1) brings performance back to about the same levels. I hope that the bug behind this can be squashed.

If you say that the image errors are trivial, then I will take your word for it.

Edit: Didee's new TGMC requires VerticalCleaner (http://home.arcor.de/kassandro/prerelease/VerticalCleaner.rar) by kassandro. Ain't the hamster wheel of upgrades fun?

JoshyD

1st March 2010, 03:40

A rough Vertical Cleaner is up on the first page, it's just the C-Code compiled, but ICC should be doing some of the SSE optimizations for us. TempGaussMC Beta 2 is at least as fast in most cases with this version of the plugin, sometimes still faster. This little filter is far from the limiting factor of the overall script . . .

Edit: The C version is faster than the ASM, odd huh?

Stephen R. Savage

1st March 2010, 04:11

Bravo, it works. It's a shame that you're the only one working on Avs64. Is there anything that those of us who can't write significant amounts of code can do?

turbojet

1st March 2010, 22:40

I see directshowsource source in the sources package but any chance of including directshowsource.dll in the binary package?

I've never been able to get squid80's directshowsource.dll working, keeps giving no function errors.

Also if you ever plan on an x64 build of 2.60 I look forward to it considering a resize only script is about 10% faster with 2.60 over 2.58 and about 15% faster then 2.58 MT.

Stephen R. Savage

1st March 2010, 23:06

DSS was included in one of the earlier copies of the Avs64 build posted here. It seems JoshyD has forgotten to include it in the latest one, so I have uploaded it here: http://www.sendspace.com/file/tm7pcf

turbojet

1st March 2010, 23:47

Thanks for the upload but unfortunately that doesn't work either. I tried both loadplugin and auto load from plugins directory and this is the result:

avs [error]: Script error: there is no function named "directshowsource"

avisource worked in avisynth64 and graphstudio64 handles the input files correctly.

Has anyone been able to use directshowsource?

Stephen R. Savage

2nd March 2010, 02:45

DSS64 works fine for me, and I used it in all my performance tests above.

JoshyD

2nd March 2010, 07:49

Whoops, sorry guys, updating the archive with direct show source, and it definitely is working for me. Don't know what other idiosyncrasies would be giving you grief when loading the plugin.

@Stephen:
I compiled a 64bit avs2avi for benchmarking and such, works fine for me, hope it does for you as well.

turbojet

2nd March 2010, 19:14

I was able to get directshowsource working. Loadplugin was pointing to a 32 bit dll.

Also I made an batch installer (http://www.mediafire.com/?zgnyfxizmim) that checks for x64 os, checks if avisynth x86 is installed, copies files to the avisynth plugins directory, and enters the registry entries. It's kind of a rough installer but it only takes one click and gets the job done. Feel free to edit at your will and/or include it with the binaries if you want.

It's ok to use avisynth x86 plugins directory for x64 dll autoloading but they need to be named different from x86 dll's to allow x86 dll's to load.

Stephen R. Savage

3rd March 2010, 00:50

@JoshyD: Thank you. I have updated the benchmarks to reflect the more accurate avs2avi methodology. avs2avi allows the accurate measurement of even very fast filters.

@turbojet: Good job figuring out the autoload thing. I thought it would be something like that.

JoshyD

3rd March 2010, 02:45

@Turbojet: Thanks for the nifty little script, I'll go ahead and stick it in the archive with future updates of Avisynth64.

@Stephen: How many threads were you allowing during those benchmarks? I'm guessing you just ran it through the normal cache, so just double checking.

Also, are there any other little pieces of code that you'd like to see work with the project? I took a look at TIVTC, and it's a bit of a beast to convert, are there any other day to day plugins that would be beneficial?

I'm thinking I might optimize some of those hot spots where a single thread gets "stuck". I think there's some speed gains hidden in removing the core program's reliance on the mmx register set. All 64 bit processors have 16 XMM registers, 6 of which are volatile across function calls, meaning we can do whatever we want with them.

The BitBlt Functions (http://en.wikipedia.org/wiki/Bit_blit) move data in 64bit chunks when we could be doing it in 128bit chunks.

I at least want to get the memory copy functions using the XMM registers instead of the MMX ones.

Stephen R. Savage

3rd March 2010, 04:09

All tests were strictly single-threaded. Neither MT() nor SetMTMode() were used, and only single-threaded filters were used.

If you can't get TIVTC, then perhaps:

Stuff I use frequently (descending order of priority):

DGDecode
GradFun2DB
dfttest
AddGrainC
ColorMatrix
FFmpegSource2
aWarpSharp (SEt version)
Vinverse (DLL version, but has working script equivalent)

Stuff I have but don't really use much (unsorted):

Average
BiFrost
ChromaShift
DeGrainMedian
Gavino's Runtime and Script extensions
RawSource
SSIQ (YV12 mod)
TComb
TTempSmooth

Stuff I use that is closed source:

nnedi/nnedi2 :(
Checkmate (needs to be reimplemented anyway)
DSS2 (dead)

Adub

5th March 2010, 23:45

Is it possible to run 32bit plugins with 64 bit Avisynth? Or will all plugins require a rewrite to work correctly with 64bit Avisynth?

kemuri-_9

6th March 2010, 14:27

Is it possible to run 32bit plugins with 64 bit Avisynth? Or will all plugins require a rewrite to work correctly with 64bit Avisynth?

x86 and x86_64 code is not allowed to intermix by the operating system.
if the plugin uses inline asm then it'll need to be rewritten...
(seeing at how most plugins inline asm instead of using an asm compiler like yasm/nasm)

if it only uses C/C++, then it can (most generally) just simply recompiled for x86_64
(this depends on how assumptive they are of things and used code that breaks when the sizeof some of the basic variable types change).

if you want to have avisynth x86 and x86_64 intermix, the available option is to use TCPDeliver to deliver frames over socket connections between them.

Stephen R. Savage

6th March 2010, 19:36

By the way, I am reporting that the version of FFT3DFilter on the first post of this thread does not work. It returns "could not load plugin fft3dfilter.dll". I have tried with fftw3.dll in system32, the application directory, and in another global/path directory.

Also, instead of making FFT3DFilter link against fftw3.dll, why not use the default name libfftw3f-3.dll, so that the DLL can be shared with dfttest (if you ever port it).

JoshyD

6th March 2010, 20:54

Also, instead of making FFT3DFilter link against fftw3.dll, why not use the default name libfftw3f-3.dll, so that the DLL can be shared with dfttest (if you ever port it).

Because I was being lazy . . . I'll change the library name to the original. I was wondering if maybe I included the wrong version of the fftw3.dll with the rar? I can't check that right now, but if you have a free second, try grabbing the original package here (ftp://ftp.fftw.org/pub/fftw/fftw-3.2.2-dll64.zip), and renaming the dll?

I had it up and running on my system . . . I'll get back to my development machine and check it out.

Also, in other fun news, I've ported the vertical resizers to use SSE registers instead of mmx, and taken it out of inline assembly. I'm almost done removing the default inline assembly horizontal routines to their own assembly functions.

You'd be surprised how much speed you can gain by just not using inline assembler and going with straight up assembly instead.

After that's all done, I think the slowest internal Avisynth function will be bitblt, and then it's on to the plugins for more optimizations.

JoshyD

6th March 2010, 20:56

Is it possible to run 32bit plugins with 64 bit Avisynth? Or will all plugins require a rewrite to work correctly with 64bit Avisynth?

The response you got earlier is quite correct, but I was wondering which plugin in particular you were interested in?

Stephen R. Savage

6th March 2010, 21:38

JoshyD, the MD5 of my fftw3.dll matches the one from the main page (Win64 version, of course).

Also, I believe the resizer functions were optimized for SSE2 in the 2.6 branch. Hopefully your optimizations won't be for nothing and can be generalized to the CVS version.

squid_80

7th March 2010, 14:52

Softwire was alive a few years ago, when 64-bit support was added: http://cvs.gna.org/cvsweb/softwire/?cvsroot=softwire

For benchmarking, that is what the "play" command of avsutil is for.

JoshyD

7th March 2010, 19:33

@Stephen

The optimizations shouldn't be all lost. The only difference is that mine aren't dynamically generated like they are in the new branch. Currently, I'm implementing them with function pointers to different resizers depending on the FIR filter size which is dependent on the filter. For example Lanczos4Resize has a FIR filter size of 8, it's possible to generate larger FIR filters, but those are rare. I'm still kicking around ideas on how to best handle these cases.

The trade here is that the actual code is larger, but since it's generated at compile time, there's no one time cost of function generation during filter instantiation. Will it really make a difference? Not really sure right now.

To give an idea of the difference in execution, I do something along these lines:

switch (plane)
{
case 2: // take V plane
cur = resampling_patternUV;
fir_filter_size = *cur++;
src_pitch = src->GetPitch(PLANAR_V);
dst_pitch = dst->GetPitch(PLANAR_V);
xloops = ((src->GetRowSize(PLANAR_V_ALIGNED)+15) / 16)*16; // Round to multiple of 16
dstp = dst->GetWritePtr(PLANAR_V);
srcp = src->GetReadPtr(PLANAR_V);
y = dst->GetHeight(PLANAR_V);
yOfs2 = this->yOfsUV;
(((INT_PTR)srcp&15) || (src_pitch &15)) ? ua_proc_uvplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur)
:a_proc_uvplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur);
break;

case 1: // U Plane
cur = resampling_patternUV;
fir_filter_size = *cur++;
dstp = dst->GetWritePtr(PLANAR_U);
srcp = src->GetReadPtr(PLANAR_U);
y = dst->GetHeight(PLANAR_U);
src_pitch = src->GetPitch(PLANAR_U);
dst_pitch = dst->GetPitch(PLANAR_U);
xloops = ((src->GetRowSize(PLANAR_U_ALIGNED)+15) / 16)*16; // Round to multiple of 16
yOfs2 = this->yOfsUV;
plane--; // skip case 0
(((INT_PTR)srcp&15) || (src_pitch &15)) ? ua_proc_uvplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur)
:a_proc_uvplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur);
break;

case 3: // Y plane for planar
break;

case 0: // Default for interleaved
(((INT_PTR)srcp&15) || (src_pitch &15)) ? ua_proc_yplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur)
:a_proc_yplane(srcp, dstp, src_pitch, dst_pitch, y, xloops, yOfs2, cur);
break;

default:

break;
}

The 2.6 branch does something along these lines:

switch (plane) {
case 2: // take V plane
src_pitch = src->GetPitch(PLANAR_V);
dst_pitch = dst->GetPitch(PLANAR_V);
dstp = dst->GetWritePtr(PLANAR_V);
srcp = src->GetReadPtr(PLANAR_V);
y = dst->GetHeight(PLANAR_V);
yOfs2 = this->yOfsUV;
(((int)srcp&15) || (src_pitch &15)) ? assemblerUV.Call() : assemblerUV_aligned.Call();
break;
case 1: // U Plane
dstp = dst->GetWritePtr(PLANAR_U);
srcp = src->GetReadPtr(PLANAR_U);
y = dst->GetHeight(PLANAR_U);
src_pitch = src->GetPitch(PLANAR_U);
dst_pitch = dst->GetPitch(PLANAR_U);
yOfs2 = this->yOfsUV;
plane--; // skip case 0
(((int)srcp&15) || (src_pitch &15)) ? assemblerUV.Call() : assemblerUV_aligned.Call();
break;
case 3: // Y plane for planar
case 0: // Default for interleaved
(((int)srcp&15) || (src_pitch &15)) ? assemblerY.Call() : assemblerY_aligned.Call();
break;
}

The real difference is in the function calling, the underlying code is really very similar. The references to assemblerY.call() or ua_proc_yplane(...vars...) execute very similar code as far as I can tell.

I checked my FFT3DFilter, and it works still. I think I may have been building it funny, I noticed some intel compiler specific include directories in my configuration, so I removed them, and rebuilt. Would you humor me and try this build? (http://www.mediafire.com/?zzz3dy0mdt2)

@squid_80
Softwire does indeed have 64bit support, but it's apparently lacking. The main team of Avisynth guys has been updating 32bit Softwire as they go, I've noticed. I guess I just feel more comfortable writing hard coded assembly than dealing with Softwire, which will probably come back and bite me later, we'll see.

I can see myself breaking down and starting to implement the same things with Softwire64, it wouldn't be much of a stretch.

Stephen R. Savage

7th March 2010, 19:57

Oh, figured it out:

Activation context generation failed for "FFT3DFILTER.DLL". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8" could not be found. Please use sxstrace.exe for detailed diagnosis.

There is also a dependency on libmmd.dll and ieshims.dll.

Edit: It loads and works fine after I installed the MSVC 2008 SP1 runtimes and a googled copy of libmmd.dll. Hopefully you can resolve the dependencies on your end as well (static linking?).

Adub

7th March 2010, 20:28

The response you got earlier is quite correct, but I was wondering which plugin in particular you were interested in?

Well, I have been considering installing Windows 7 x64 bit for some time as the website I run (Adubvideo (http://adubvideo.net) gives a number of tutorials, and on some of these my readers have been having issues when running BD Rebuilder on Windows 7.

Since I'm a bleeding edge kind of guy, I though, let's go 64bit! But I wanted to be sure that I could still get the most out of my current encoding chains.

I currently don't have a particular filter in mind, but then I haven't gone through all of my plugins to check which ones are compatible and not. Judging as more and more support is being added for 64 bit, I'll think I will go ahead and make the switch in the next few weeks.

I'll be back with plugin suggestions soon after that, I'm sure! ;)

And thank you so much for your hard work! It is greatly appreciated right now with the big switch to 64 bit operating systems and processors.

JoshyD

8th March 2010, 01:18

Yeah, statically linking would be pretty helpful, somehow I missed that one when setting my build options. It's fixed now, I think we're good to go

JoshyD

9th March 2010, 21:07

New build is up on the main page, please download it and let me know if I've missed any oddities in my test cases.

Changes are listed on the front page, there's some nice speed gains to be had in the main dll. Running single threaded TempGaussMC beta2 through avs2avi with a null output and using an old home movie recorded in DVSD (720x480, 29.97fps, interlaced, bff) as the source:

Avisynth32: 5.05fps
Avisynth64: 6.09fps

Hooray, a whole extra frame every second! It may not seem like much, but for a long encode (each run took ~40mins, give or take depending on the version used), an extra frame every second, can shave a significant amount off your total encode time.

Add some good ol' SetMTMode(2) into the mix, and you've got a larger gap (same test):

Avisynth32: 13.79fps
Avisynth64: 17.66fps

Still not setting any speed records, but with a slow script (that produces great results) I'll take my speed gains, however minor they may be. Script dependent, I'd say speed increases are in the 15% to 20% range on average.

Stephen R. Savage

9th March 2010, 22:27

Quick tests:

Decode 640x480 Xvid (MS MPEG-4 Decoder, 500 frames)
32-bit: 113.27 fps
64-bit: 118.20 fps
Relative Speed: 104.4%

Spline36Resize 16x Enlarge (500 frames)
32-bit: 18.06 fps
64-bit: 31.61 fps
Relative Speed: 175.0%

TempGaussMC beta 2 (EEDI2, 100 fields)
32-bit: 3.42 fps
64-bit: 3.72 fps
Relative Speed: 108.9%

MDeGrain3 (200 frames)
32-bit: 5.96 fps
64-bit: 6.54 fps
Relative Speed: 109.7%

AAA (100 frames)
32-bit: 2.17 fps
64-bit: 2.20 fps (MTMode = 1)
Relative Speed: 101.3%

All tests were run three times and averaged. I am not seeing the performance increases in TGMC that you are, JoshyD. However, resize performance is definitely up! The caching bug related to AAA is still not fixed. I suspect this bug is sapping performance out of other scripted filters (read: TGMC) as well. Note that the SetMTMode hack is suboptimal, as it costs performance when the caching bug is not in play (e.g. a simple source+resize script).

turbojet

9th March 2010, 22:59

Thanks for the new version however it crashes on Athlon II. Here's the windows error codes.

Problem signature:
Problem Event Name: APPCRASH
Application Name: x264_x64.exe
Application Version: 0.0.0.0
Application Timestamp: 4b8c1206
Fault Module Name: avisynth.DLL
Fault Module Version: 2.5.8.5
Fault Module Timestamp: 4b9681a6
Exception Code: c0000005
Exception Offset: 0000000000005528
OS Version: 6.1.7600.2.0.0.256.1
Locale ID: 1033
Additional Information 1: 9eab
Additional Information 2: 9eabb149e34b0e02564736c484278831
Additional Information 3: 6bcd
Additional Information 4: 6bcdaf28393e1989487185c90748dcec

I noticed the new dll is 4 MB while the older one that works is 800 KB.

Also since all 64 bit filters I could find are named identical to the 32 bit counterpart I changed the install script to use plugins64 directory. You can grab it here (http://www.mediafire.com/?zgnyfxizmim)

Another thing is as far as I know there's only one haali media splitter build that's x64 and handles vc-1 and it's very difficult to find but I uploaded it here (http://www.mediafire.com/?yy5nimzdwed). Up to you if you want to post it in the original post in case people aren't able to use HMS x64 or report issues with VC-1 (from the latest official release).

Lastly about filters I heavily use TIVTC so it's unfortunate that's not easy to convert. The other ones I use every once in awhile are:
Decomb v4 (http://neuron2.net/decomb/decomb410b4.zip) (more effective ivtc then v5 but nowhere near tivtc, squid's is v5 only)
LeakKernelDeint (http://avisynth.org/warpenterprises/files/leakkerneldeint_25_dll_20050118.zip) (fast, simple, sharp deinterlacer)
RePAL (http://avisynth.org/warpenterprises/files/repal_25_dll_20030523.zip) (very effective for handling pal sources that were blended for ntsc dvds)

JoshyD

10th March 2010, 00:14

@Stephen: Let TGMC (and all the filters, really) have a few thousand frames to work with, 100 doesn't give the caching mechanisms in Avisynth (nor the computer as a whole) time to get filled and ready to go. All that data has to be pulled in closer and closer to the processor before any computationally intensive algorithms can really shine. If they don't have the needed data close at hand, you're going to be memory latency limited rather than compute limited. I've been running tests with TGMC and my own personal builds to collect usage data in the program. That being said, using a larger sample can sometimes accentuate the differences between the two versions. For example, a 2002 frame sample produces slightly more differences than if I just let it run though the first few hundred frames.

Avisynth 32: 5.54fps
Avisynth 64: 6.60fps
Relative increase: 119%

The caching bug is annoying, can post the exact script and file for AAA that you're working with?

@turbojet: That's a memory access error, can you post the script that you were running when that occurred? A short sample of the clip you were using would be useful as well, I need to get pitch, height, width, etc info from it. I haven't coded any instructions that are incompatible with your processor, however, I may very well have mucked up my memory alignment access requests. I think it probably occurred in the horizontal resize function, but can't be certain.

The DLL linked here is ridiculous in size because of the compiler that generated it, and the options I allowed. Intel's C++ compiler will generate a specific code path for every Intel processor p4 and newer. At runtime, the code CPUID's your processor, and if you're lucky enough to have the your VendorID = 'GenuineIntel' you'll get a special set of the code optimized for your particular processor and its idiosyncrasies. Therefore, extra code for those processors, in addition to some statically linked code from OpenMP, balloon the size of the DLL.

Don't worry about AMD processors though, the base code path is for any processor that has SSE3 or newer. While Intel's compiler won't auto-vectorize using SSE4 instructions for AMD processors, it will still give them the benefit of data operations that can be performed with SSE3 and any older set of SIMD instructions.

turbojet

10th March 2010, 00:51

Source is a 1920x1080 m2ts, script is: DirectShowsource().AssumeFPS(24000,1001).LanczosResize(1280,720)

It's definitely an issue with resize, if I don't resize it works without crashing. I tried bilinear, point, bicubic resize and all crashed.

JoshyD

10th March 2010, 01:35

@turbojet

Would you try the latest release on the first page to see if that cleared up any of the problems?

If that doesn't, would you try the following two procedures:
1. Only resize on the the vertical axis
2. Only resize on the horizontal axis

Hopefully, only one of those will crash the program, if at all.

I'm guessing DirectShowSource is giving you a yv12 stream?

turbojet

10th March 2010, 02:11

With same 1920x1080 source
LanczosResize(1280,720) - crash
LanczosResize(1920,720) - crash
LanczosResize(1280,1080) - works

yes yv12

JoshyD

10th March 2010, 03:43

turbojet

10th March 2010, 06:31

kemuri-_9

10th March 2010, 14:16

Every source crashes instantly with horizontal resize so a source I don't think would help and I'm afraid it's a cpu instruction issue. Some things that might help is x264 doesn't use SSE3 on this cpu, this is what it uses: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT. Also a few months ago I was testing icc x264 builds and found some that worked and some that didn't. I believe the ones that didn't use -QaxSSE3 during the compile but I'm not 100% on that.

what cpu do you have again (codename preferred)?

SSE3 is barely used in x264, the majority of the SSE3 related asm actually uses SSSE3.
that being said, x264 does have some SSE3 asm but it only uses these for CPUs that flag Cacheline64, which is only on intel processors.

tl;dr
AMD processors never see SSE3 getting used in x264 even if they have it.

Stephen R. Savage

10th March 2010, 17:11

@turbojet
I can't seem to recreate the crash. I specifically wrote the vertical resize functions to check for memory alignment before executing. I hope this isn't processor specific, that would be a bummer. Checking for instruction support, your Athlon II should have all the goods to do the resize correctly. Any chance you could snip a few frames (100?) of the source and post it somewhere so I can investigate further?

@Stephen
That cache bug is annoying . . . but interesting. It seems that it also exists in SEt's 32bit build of avisynth 2.5.8 as well, perhaps something got all strange when the MT mode was hacked to be supported?

Here is AAA.avsi in full:

function AAA(clip input, int "type", bool "mask", bool "chroma")
{
ox = width(input)
oy = height(input)

type = default(type, 1)
mask = default(mask, true)
chroma = default(chroma, false)

gscale = chroma ? input : Greyscale(input)

aa = type >= 2 ? nnedi2_rpow2(gscale, rfactor=2) :
\ type == 1 ? TurnRight(gscale).EEDI2(field=1).TurnLeft().EEDI2(field=1) :
\ PointResize(gscale, ox * 2, oy * 2).TurnRight().SangNom().TurnLeft().SangNom()

edge = mt_didee(aa).Spline36Resize(ox, oy, -0.5, -0.5, 2 * ox, 2 * oy)
ds = Spline36Resize(aa, ox, oy, -0.5, -0.5, 2 * ox, 2 * oy)
maskmerge = mask ? mt_merge(input, ds, edge, U=1, V=1) : ds

return chroma ? MergeChroma(maskmerge, ds) : MergeChroma(maskmerge, input)
}

function mt_didee(clip input)
{
mask = mt_logic(mt_edge(input, "5 10 5 0 0 0 -5 -10 -5 4", 0, 255, 0, 255),
\ mt_edge(input, "5 0 -5 10 0 -10 5 0 -5 4", 0, 255, 0, 255), "max").Greyscale().
\ Levels(0, 0.8, 128, 0, 255, false)
return mask
}
I just call it as
DirectShowSource("source.avi")
AAA()
adding "chroma=true" makes the cache bug even worse. I don't use MT myself, so this cache bug is quite a bummer. Hope you can find it, even if it's not a bug in your own code. Perhaps it will even turn out to be an intractable design flaw (argh).

Also, many older AMDs do not support SSE3, but only up to SSE2 (slowly).

Wilbert

10th March 2010, 17:32

Also, many older AMDs do not support SSE3, but only up to SSE2 (slowly).
My old Athlon XP doesn't support SSE2, but only iSSE (or perhaps SSE dunno).

Stephen R. Savage

10th March 2010, 17:48

According to Wikipedia, the oldest AMD64 CPU (Opteron, 130nm) supports SSE2. However, turbojet's Athlon II appears to support SSE3, so who knows...

JoshyD

10th March 2010, 18:32

@Stephen
You're quite correct, which is why I was confused. I don't think his processor agrees with loading the values of some of the arithmetic functions straight from memory. For now, I'm going to band-aid the code paths to use MMX if a non-compatible CPU turns up. I had totally forgotten that the Athlon64's only supported SSE2. I was happily thinking that many of the feature checking before function execution were going to go away because x64 processors generally have the latest and greatest when it comes to SIMD instructions. Looks like that's not the case. I've got an old Athlon64 I rarely use, so I guess I'll be turning it on to run test vectors before any future code gets loose.

It's weird because he can use the horizontal resize functions which make heavy usage of 128bit memory transfers to set up their workspaces. SSE2 doesn't have official listings for movdqa or movdqu instructions, but those are most certainly used when resizing YV12 or YUY2 horizontally.

The cache issue remains open, and I'm able to perfectly recreate it using any source. Looking intensively over a diff with the core Avisynth files hasn't turned up anything of interest yet. This test case seems to be the only one that really highlights the problem. Running other filters, internal and external, with and without a SetMTMode command doesn't give such a stark contrast in performance. I'm wondering a) why this filter combination is a showstopper and b) what script environment variables aren't set unless you allow a MT mode to be set. The largest difference I could find is:if ((env->GetMTMode(false) > 0) && (env->GetMTMode(false) < 5)) {
filter_graph = new CacheMT1(new Distributor(filter_graph, env), env);
}
else {
filter_graph = Cache::Create_Cache(AVSValue(filter_graph), 0, env).AsClip();
}

This occurs on script instantiation. I'll keep looking. On a sidenote, building and using the current 2.6 allows MT mode to be set, but the performance is abysmal. All non-MT functionality is perfect though.

@kemuri-_9
Thanks for the heads up on the x264 methodology of choosing instructions based on cacheline size. This makes me wonder if turbojet's CPU will return "true" when asked if it supports SSE3. If that's the case, then I'll have to add in code (yank it from the x264 cpuid functions) that checks for cache line size, and goes about its business appropriately.

Looking at the 2.6 CVS, it dynamically assembles almost identical code to what I have written for the vertical resizers. It does so based upon a check for SSE3 and SSSE3. If it's the case that Athlon II's and their brethren say they have SSE3, but fail when executing these SSE ops, the main code branch may hit a similar snag.

@turbojet:
Would you mind running CPU-z (http://www.cpuid.com/download/cpuz/cpuz_153.zip) and telling me what feature flags your CPU reports? If it says SSE3, then dang, more code to write.

EDIT:

Caching bug squashed. Turns out, EEDI2 identifies itself as never wanting a cache, so when your script asks for a frame generated by EEDI2, it goes all the way back and generates it again.
if (h_policy == CACHE_NOTHING) { // don't want a cache. Typically filters that only ever seek forward.
__asm mov rbx,rbx // Hack! prevent compiler from trusting ebx contents across call
return childGetFrame(n, env);
}

That childGetFrame executes more times than needed when your AAA script is run. This is the cause of the huge slowdown. Can anyone think of an instance where we would universally never want to save the previously generated frames? Check the main page for an update, this also *tries* to address the problems with older processors, but I don't think it's all the way yet.

turbojet

10th March 2010, 23:15

It's an athlon II 620. CPU Instructions are: MMX(+) 3DNow(+) SSE(1,2,3,4A) x86-x64, AMD-V which I'm pretty sure is identical to Phenom II. At least linux /proc/sys/cpu has identical flags.

Avisynth 2.6 Alpha 2 works fine here and if a dropback is needed wouldn't it be better to drop back to SSE(2) instead? IIRC that's all the further 2.6 branch goes to and it's a significant speedup

Stephen R. Savage

11th March 2010, 00:04

Updated results for AAA (200 frames):

32-bit: 1.85 fps
64-bit: 1.90 fps
Relative Performance: 102.7%

A bit disappointing, since I recalled EEDI2 being 10% faster and masktools2 similarly faster. How did you fix the caching bug? Did you enforce a cache for all filters? How does the original Avisynth code handle this?

Also, here's to hoping that you manage to get TIVTC ported one day, as it's one of the Greatest Filters Of All Time™.

JoshyD

11th March 2010, 03:45

@Stephen
The quick fix was to just *allow* a filter to be cached. EEDI2 was requesting not to be cached at all for some reason or another. Before, if the policy was not not cache a filter, it would go to the filter's get frame function, and return that, and no cache related code would be executed. By disallowing the insta-return and allowing it to also check / insert the frame into the cache, AAA starts finding the frames EEDI2 previously generated, and uses those, rather than making EEDI2 do the work all over again.

I'm not sure what the difference is between SEt's build and the standard build that creates the slowdown. The MTModes are hacked in to the main program, so there's some incongruities in the code. I think the internal caching mechanism is being polished little by little for 2.6, so this will be cleared up in the end. For now, the fix uses a little extra memory in certain corner cases. In general, a cache irregardless of whether or not the filter was written to only seek forward isn't a bad idea. You can write a script to access the source in any pattern you like really. If the accesses repeat, the script will be faster.

Also, a comment on the speed differences in AAA, over 3 runs of 250 frames, I have:
32-bit: 3.28fps
64-bit: 3.69fps
Relative performance: 112.5%

Did you grab the latest build of EEDI2? It used to use the same code for memory copying as Avisynth, which I wrote to work on processors with 128bit registers, instead of 64. The memory copy alone adds a little speed bump.

TIVTC is a beast because of the hodge podge of code it consists of. It also intermingles inline asm with compiler intrinsics as a means of using the XMM registers in some cases. It can get a little ugly. If I can figure out what's causing turbojet's crash, I'll re-examine TIVTC.

@turbojet:
The latest grab of the 2.6CVS has vertical resize code that checks for SSE3 support and then uses the same combination of instructions I do. I wish I had a similar machine to test on, so I could trace during run time. When I said fall back to MMX, I meant the mmx registers. They're half the size of the SSE registers, but technically, iSSE instructions use them. I'd drop back to that code in the case of an incompatible processor. The conundrum is why is your SSE1-4A supporting processor balking at the code? I also wonder what x264 is using to red flag your CPU to restrict it to SSE2, perhaps the vendor ID?

Do the older versions of the DLL let you resize? I hadn't changed ANY of the resize code at that point. I guess that's a good sanity check point. Try this one (http://www.mediafire.com/?cnzdxvzkw0m) or the fail safe is this build of the source. (http://www.mediafire.com/?ykdjez2eomy) If that doesn't work, there's something deeper behind the problem.

Has anyone been able to get this to run correctly on AMD cpu's?

kemuri-_9

11th March 2010, 04:48

I also wonder what x264 is using to red flag your CPU to restrict it to SSE2, perhaps the vendor ID?

look at common/cpu.c (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=common/cpu.c;hb=HEAD) and common/x86/cpu-a.asm (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=common/x86/cpu-a.asm;hb=HEAD)

osgZach

11th March 2010, 15:43

Does anyone have 32 vs 64 bit performance numbers for TempGaussMC_beta2 ? (assuming it runs.. Stephen mentioned TGMC but not which version he's using, I don't think)

Stephen R. Savage

11th March 2010, 18:36

@JoshyD: I'm using the copy of EEDI2 from the first page, which is dated to 2/19/2010. If there is another version, I am not aware of it.

osgZach

11th March 2010, 19:19

I was referring more to the AVS(i) script itself. But I'm guessing we're talking about the same thing really.

Either way. I average about 2fps give or take some fractional ups and downs. Takes about 4h:45m to do a 22m:48s clip

Core 2 Duo @ 3.2ghz

What kind of fps do you see on your machine?

And while I have your attention.. any way to get it entered either into the Fieldhint(blah..()) or replace that line entirely, during Yatta AVS generation ?

Sucks to have to manually swap it out on 50 different project files on average :p
But I suppose nothing a find/replace script can't fix. Still learning bits and pieces about YATTA, but since writing my Find30fps tool it's been so much less trouble.

JoshyD

11th March 2010, 19:59

Temp Gauss Beta 2 were the runs Stephen and I were performing, single threaded, for me enjoys a healthy ~20% speed increase, multiple threads speeds up the process by a larger margin. Here are some sample numbers for a 4000 frame run of a dvsd (720x480, 29.97 fps, interlaced bff) source through the script using avs2avi, no compression, outputting to null:

32bit Avisynth: 5.06fps
64bit Avisynth: 6.07fps
Relative speed: 119.96%

Threading the script with SetMTMode(2,8):
32bit Avisynth: 14.47fps
64bit Avisynth: 18.05fps
Relative speed: 124.74%

Tests were run with a Core i5-750 (4 cores, no HT) @ 3.71GHz. The eight thread creation request keeps all 4 cores constantly churning at 100%. Requesting less threads reduces total CPU utilization, with 4 threads giving ~50% total usage. Your performance will vary based upon your system setup, obviously.

The versions of Avisynth were both based on SEt's 2.5.8 build with multithreading enabled, for as much of an apples to apples comparison as possible.

Generally, it seems safe to assume ~15-20% performance increase, dependent upon which plugins you want to use.

Vertical Sharpen was ported specifically because of it's use in TGMC beta 2. The same goes for RemoveGrain and Repair. MaskTools2 and MVTools2 were ported because they're so darned useful. EEDI2 was ported to fill the void of a good 64bit deinterlacer.

The 32 bit versions of these plugins were compiled by myself, with some tweaks here and there as I went. I've been "rolling my own" versions of these for a while now, the 64bit port was an extension of this hobby.

osgZach

11th March 2010, 20:10

Thanks for your response.

Those are certainly some nice numbers. I wasn't even aware you could run it multi-threaded either. Frankly the setup process for MT related stuff scared me way, was afraid I would boink something :p
Right now I only exclusively use TempGaussMC_beta2, the only other filter being TDecimate, and any other stuff YATTA deems necesarry in the generated AVS. I am encoding to HuffYV12 and then filtering later. So hey maybe I'll get something like 5 or 6 fps ? LOL :p

Perhaps I will go back to the initial post and see if I can follow the steps to do all of this..

Although as far as MT goes.. I only have 2 cores, so I wouldn't expect huge numbers like you got (but really impressive), but I think the x64 single threaded boost might be pretty big as well.

Is there a chance this will all be available vial a one-click installer at some point?