64 bit avisynth?

[Blue_MiSfit]

Hello all,

I am just wondering if there are any plans to release avisynth for the Windows XP x64 edition beta. XviD and VirtualDub both have 64 bit branches now, and AviSynth is the only missing tool :)

If anyone has any information regarding this please let us all know!

Thanks
~misfit

[sh0dan]

MS has decided to ditch the current method of doing inline assembler, so basicly most assembler code has to be redone. :(

[Blue_MiSfit]

ouch.

[Richard Berg]

Intel's 64-bit compiler supports it (and has fewer bugs, from talking to several people), but their builds tend to be much slower than MS or GCC on AMD chips.

[Cyberia]

Originally posted by Richard Berg
Intel's 64-bit compiler supports it (and has fewer bugs, from talking to several people), but their builds tend to be much slower than MS or GCC on AMD chips. I thought I just read somewhere that the Intel compiler does not detect AMD processors having MMX/SSE/SSE2. It's a sneaky thing to do to make AMD look bad. There was a workaround too to force the optimizations anyway and then AMD performed very well indeed.

[d'Oursse]

cyberia: i think that it's here :

http://groups.google.ca/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&group=comp.arch&selm=a13e403a.0402091438.14018f5a%40posting.google.com

[squid_80]

Wouldn't the assembly code need to be redone anyway?

[708145]

Originally posted by squid_80
Wouldn't the assembly code need to be redone anyway?

not completely, only the parts that would benefit a lot.

and yes: I'm looking forward to both Win64 and Lin64 versions of avisynth!

bis besser,
Tobias

[squid_80]

Originally posted by 708145
not completely, only the parts that would benefit a lot.


I mean any assembly code that uses memory addresses has to be redone, the 64-bit extended GPRs (rax, rcx etc.) must be used instead of the 32-bit registers (eax, ecx etc). So even if a compiler did support inline assembly, it would all still probably have to be modified. Not to mention other issues like microsoft saying MMX can't be used, certain instructions not being available in 64-bit mode...

[Sharktooth]

that's only an OS issue. the processor itself supports all mmx and x87 instructions in 64bit mode.
btw, you're right a big part of ASM should be rewritten but there are some tools on the AMD website that could help the transition a lot...

[squid_80]

Originally posted by Sharktooth
btw, you're right a big part of ASM should be rewritten but there are some tools on the AMD website that could help the transition a lot...
If it's that Convertsse2instrin perl script you're talking about, it sucks. I think they've removed it, and with good cause.

The instructions not available are things like "mov ah, [r8]" and a few undocumented opcodes like salc. Don't know if avisynth uses any of them, most are easy enough to work around anyway.

[JRepin]

Hi,

Has there been any progress on making 64-bit version for AMD64 possible? If not, when can ve expect it? Are there any plans for this?

Thanks for answers!

[Richard Berg]

If you read the above comments, you'll see we're not optimistic. Actually, if this blog is correct, the situation is even worse:

Regarding the (ick) comments about MMX: Don't use MMX on x64 [I don't think the ML64 assembler even supports it!]. I haven't yet seen a single scenario where MMX code runs faster than SSE2 code. The primary reason I've heard that people want to use MMX is because they already have a bunch of x86 ASM code using MMX. I have 1 word for you: rewrite. Chances are, if you don't rewrite it, your asm code will be completely wrong & broken, due to ABI restrictions - you won't discover this until you find yourself staring at a broken stack dump, or a terminated process, due to failed exception handling.
http://blogs.msdn.com/freik/archive/2004/11/06/253291.aspx

Honestly, I don't see how porting Avisynth to AMD64 would help, since all the time-sensitive portions are already vectorized. Some of the long script-functions being written by guys like Didee would benefit from large amounts of virtual memory, but not until machines with 2GB+ of RAM are common.

[JRepin]

Well you need a 64-bit app to use it from other 64-bit apps. You can't use 32-bit app. And avisynth is currently the only missing link in the 64-bit chain. Oh and some of us alreays have more then 2 GiB of memory

[708145]

Just to brag a bit:
dual opteron 250 with 16GB RAM here!

Eagerly awaiting 64 bit linux avisynth :p

bis besser,
Tobias

[Shinigami-Sama]

considreing I'm planing on buying an amd64fx mobo next
they would be nice to see
and iirc hte amd64 version of mmx sse1/2/3 ect. are slightly more effiacnt that intels;
but that might be flawed info on my part
but I love to see this come out before I buy hte amd64fx

[Richard Berg]

AMD K8 chips are slightly more efficient at MMX/SSE in that some of the latencies are lower, yes. However, if the instructions are ordered well enough that the P4's decoder can keep up, it's not enough extra efficiency to make up for the P4's clockspeed advantage.

[Shinigami-Sama]

AFAIK...
the AMD 64fx has a built-in ram manager as well
so that bumps up tranfer speeds of everything
and I guess I should clearify a bit more
I ment the 64fx;s that are coming out soon or just came out their new err 967 pin one
iirc I read that its latency was 80% lower than intel;s lastest P4
but I read that about a month ago I can;t rembmer hte link so I can't check though:(
but also
I read somewhere that intel uses two 64bit halves for sse and AMD uses a real 128bit channel
if so would that not make it more efficnat?
because 64bit+64bit=65bit seeing as how every bit you add you double hte infomation
but I might be off due to sleep deprivation though
but I still prefer amd over intel so this would give me a greater incentive to get hte ubernice 64fx when I get hte money

[squid_80]

Sorry to bring up this old thread again, but is anyone doing any work on porting to x64? I've done some basic work, got most of the core stuff ported and just finished an x64 version of dgdecode to go with it, I just want to make sure I'm not wasting time doing work that's already been done by someone else.

[sh0dan]

To my knowledge there isn't anyone else working on 64bit 2.5 based AviSynth. I'd love to help, but I don't have the platform available (or rather the OS available). Are you using VS .net 2003?

[squid_80]

Unfortunately, yes - VS.net 2003 with an old release of the winddk. I'm having problems with exception handling, Avery Lee tells me the latest VS 2005 beta fixes them but unfortunately I don't have an MSDN subscription so I can't get it. I figure I might as well wait for the retail version to come out.
Re the inline assembly code, which do you think would be a better way to go:
1. Changing it to intrinsics so it's not processor specific but still reasonably fast
2. Break the code out into seperate files, modify to x86_64 assembly and compile with yasm (or ml64, I guess). I prefer this method since you get more control over the hardware, but it's very non-portable. Plus you can use the MMX registers, although that's a bit of a grey area.

[sh0dan]

Isn't intrinsics MS specific? AFAICT, YASM should be portable to other x86-64 based systems, except that

- I'm still hoping for SoftWire to become 64 bit aware (http://sourceforge.net/forum/forum.php?thread_id=1095618&forum_id=184692). That would be a great solution.

[tsp]

It seems as if someone did make a 64 bit build. I don't know if it's 3.0 or 2.5 though.
http://www.planetamd64.com/index.php?showtopic=5013

[d'Oursse]

i don't think it's for 3.0 ;)

[Wilbert]

@squid_80,

1) On which build is AviSynth64 based?

2) Could you also release its source?

[squid_80]

@shodan: Softwire support would indeed be a great solution. When I said using yasm would be less portable I meant that it would be possible for someone to break the assembly unknowingly by changing the C code... I guess it's still not very clear what I mean.

@tsp: Recognise the planetamd64 user's name? ;)

@Wilbert: It's based on 2.55. The source is nothing special, it's mostly the same except for the resample code which got hacked up something terrible (putting pointers in an int array is not a good thing). It's available at http://home.iprimus.com.au/ajdunstan/avisynth64src.zip

[Richard Berg]

Avery Lee tells me the latest VS 2005 beta fixes them but unfortunately I don't have an MSDN subscription so I can't get it.
Beta 2 will be available to everyone very soon.

[squid_80]

Originally posted by Richard Berg
Beta 2 will be available to everyone very soon.

Good to hear, should fix the broken stack unwinding I'm getting. Hopefully the license will allow for redistribution.

[IanB]

Dear all,

As discussed (PM's) with Wilbert I have added Andrews source to the CVS repository with the branch Avisynth64.

Take a peek at Avisynth64 CVS branch (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/?sortby=date&only_with_tag=Avisynth64#dirlist)Originally posted by squid_80
@Wilbert: It's based on 2.55. The source is nothing special, it's mostly the same except for the resample code which got hacked up something terrible (putting pointers in an int array is not a good thing). It's available at http://home.iprimus.com.au/ajdunstan/avisynth64src.zip Regards
IanB

[Wilbert]

So, anyone who wants to help squid_80 with porting (resizers except in YV12, color conversion routines and other stuff) can send us a PM.

Btw, dgdecode 1.2.1 is already done! But, it's the only plugin though ...

[squid_80]

Thanks guys. I'm struggling to get a x64 build of 1.1.0-beta2 xvid done at the moment (weird problems decoding b frames when compiled with the new PSDK, grrr) but as soon as that's sorted I'll get back to work on avisynth64. First task is to clean up all that sizeof(BYTE*)/sizeof(int) mess. I dunno why I didn't use a #define in the first place.:D Also I can't remember what I did with the Bitblts, but I bet they're less than optimal.

I guess if people want to start suggesting plugins (that have source available) that they would like to see done, here is as good a place as any. But be prepared to be patient (unless you want decomb - it's already half done).

[708145]

Originally posted by squid_80

I guess if people want to start suggesting plugins (that have source available) that they would like to see done, here is as good a place as any. But be prepared to be patient (unless you want decomb - it's already half done).

Great work.

But I'm confused: Is avs64 the same as avs3 ?
And which (if different) supports linux64 native?

About the filters: Just keep on porting :p Sadly for most filters I use the source is not published :(

bis besser,
Tobias

[Wilbert]

But I'm confused: Is avs64 the same as avs3 ?
Nope. (avs64 is based on 2.5.5, and avs3 is totally rewritten ...)

And which (if different) supports linux64 native?
The former for sure doesn't. I don't know about avs3 (don't think so, but perhaps bidoche can comment here ...).

[d'Oursse]

the 3 asm files of avs3 are not written for 64bits processors.

for a 64 bits version of avs3 on linux, just use a version of gcc that produces 64 bits executable/library

[708145]

Originally posted by d'Oursse
the 3 asm files of avs3 are not written for 64bits processors.

for a 64 bits version of avs3 on linux, just use a version of gcc that produces 64 bits executable/library

:D So avs3 compiles (without asm) on linux64? Awesome! :D

Now it just has to be finished...

bis besser,
Tobias

[pogo stick]

Squid, thank you for your build. I wanted to try this, but it didn't work for me. I did everything you wrote: copied avisynth.dll, clicked .reg, rebooted, used dgdecode64.zip, but VirtualDub gives me an error.

AVI Import Filter error: (Unknown) (80040154)

Did I forget about something? :confused:

[squid_80]

You are using windows x64 and veedub64.exe, correct?

[pogo stick]

Yes, correct.
By the way, your XviD build is working.

[squid_80]

Not really sure, sounds like avisynth.dll can't be found. Make sure it's in the right /windows/system32 dir (using a 32-bit file explorer can fool you), if that doesn't work try putting it in the same directory as veedub64.exe.

[pogo stick]

If I copy avisynth.dll to C:\WXP64\system32, the same file appears in C:\WXP64\SysWOW64. If I delete it from system32, it disappears from SysWOW64.
Putting avisynth.dll in veedub64.exe directory doesn't help.

[squid_80]

Then something is wrong; sounds like some 32-bit process is making you see /wxp64/sysWOW64 as /wxp64/system32.

[pogo stick]

Oh, I get it. FAR manager (32 bit) was the one who tricked me. I should look for 64 bit file managers.

Now it's working. :)

[Wilbert]

Ok, I've had a look at avisynth64's resizing discrepancies and here's the lowdown:

I'd missed a rounding calculation in the vertical resizing, this has been fixed (also vertical resizing should work for all colorspaces, not just YV12).
There was a bug in horizontal resizing converting dwords to words. Also fixed.
Now PointResize and BilinearResize produce the same output as a 32-bit build. But Bicubic, Lanczos and Lanczos4 can still produce values that vary by a range of about 4. Since the co-efficients used for resizing are produced using lots of floating point divisions (lanczos uses sine as well), I'm guessing there's small accuracy differences due to the compiler using different methods. This is probably something that is more or less unavoidable. The question is which has better accuracy, 32 or 64-bit? Or will anyone be able to see the difference? ;)
New avisynth64 is in the usual place: http://home.iprimus.com.au/ajdunstan/avisynth64.zip



http://forum.doom9.org/showthread.php?p=666155#post666155

[squid_80]

Now that I think about it, I'm fairly sure what I wrote above (wilbert's quote) is wrong - it's more likely the coefficients are being calculated correctly but being overwritten/stored incorrectly. I'm guessing this because
a) only horizontal resize seems to be affected and it stores pointers (64-bit) in the same array as the pixel values. Probably the pointers are overwriting something or the array is overflowing.
b) Do point and bilinear use less taps than the others? Again this would indicate the array is not big enough due to the larger pointers.
Anyway, I'll work on it some more and see what happens.

Also using version() as a source to check that 2 different copies of avisynth produce binary compatible output is really dumb. I can't believe it took me two hours to realize that.

[squid_80]

I'm still trying to find how these discrepancies are sneaking in and seem to be getting pretty close. But can someone with intimate knowledge of the YV12 resampling algorithm please explain something to me about this code (from resample.cpp, starting at about line 292):
x86.paddd(mm3,mm7);
x86.paddd(mm4,mm7);
x86.psrld(mm3,14);
x86.paddd(mm5,mm7);
x86.psrld(mm4,14);
x86.psrld(mm5,14);
if (isse) {
x86.pshufw(mm3,mm3,(char)248); // 11111000
x86.pshufw(mm4,mm4,(char)143); // 10001111
x86.pshufw(mm5,mm5,(char)248);
x86.packuswb(mm3, mm3);
x86.packuswb(mm4, mm4);
x86.packuswb(mm5, mm5);
x86.por(mm3,mm4);
} else {

I'm wondering why mm3 and mm4 don't have the unneeded bits stripped away before the or operation. Near the top of the code the doubles held in the registers are shifted right by 14 bits, leaving 2 unknown bits in the high word. If these 2 bits aren't 0, they can survive the shuffles and packs and end up being or'ed into the final result. Or have I missed something?

[IanB]

Squid,

This code is the ultimate abuse of mmx arithmetic. The code that preceeds the code shown above that does the pmaddwd's and sums into mm3/mm4/mm5 has very carefully crafted coefficients from the generator code that make sure the sum doesn't exceed 16384*255*255. The great leap of faith is the way pixel pairs are treated as a single 16 bit word for much of the arithmetic. I still can't convince myself there is no carry across pixel pairs, but the code works.

To help understand the number ranges track backwards from the output "movd [eax],mm3" and make the assumption that each byte is between 0 and 255 and each deriving word or dword is ranged not to violate this, particularly the packuswb's.

Regards
IanB

[squid_80]

If there is no carry, then shouldn't this code perform the same:
if (isse) {
x86.pshufw(mm4,mm4,(char)248); // 11111000
x86.pshufw(mm3,mm3,(char)143); // 10001111
x86.pshufw(mm5,mm5,(char)248);
x86.packuswb(mm3, mm4);
x86.packuswb(mm5, mm5);
x86.psrlq(mm3,16);


(swaps the way mm3 and mm4 are shuffled, packs them both with one instruction into the centre of mm3 then shifts the results into bottom dword)

This gives different results to the existing code (try it on bright clip).

[IanB]

Hmmm, re-evaluating what I said, the sum limit should be only 16384*255. The pixel pairs stuff is total bull, I forgot about the byte to word explosion into the temp array.

For an excess to occur the coefficients have to sum to at least 16417, thats a lot of rounding error, or be negative (shudder) but I can't say it's impossible. Also there was that problem with zero coefficient sum that I was never completely satisfied with.

I've been putting off looking at this code because it is so devious and I was going to do an SSE2 version when the chroma planar alignment came of age. Research and testing required.

As for better code, this is how it should be, 9 isse/10 mmx -> 4 mmx. x86.packssdw(mm3, mm4);
x86.packssdw(mm5, mm6);
x86.packuswb(mm3, mm5);
x86.movq(qword_ptr[eax],mm3);IanB

[squid_80]

I wasn't really giving an example of better code, just a different method that should give the same answer IF the 32-bit sums are < 2^30. But it gives different results. In fact the softwire isse, softwire mmx, assembler isse and assembler mmx algorithms all seem to give slightly different results. Mainly because they all assume the two MSBs of the sums are 0. Same goes for your code sample using packssdw; values that should be 0 seem to be coming out as 0xFF due to the dword->word conversion saturating to 0x7FFF and the following word->byte conversion saturating that result to 0xFF.

I think I'm going to have to wade into this with the debugger.

[IanB]

And you have given me the answer. Some of the coefficents are negative, particularly from Lanczos, which given the right image, ones with sharp black/white transitions, will have some ringing which may overshoot negative.

The psrld's should be psrad's so that negative results stay negative and get saturated to zero.

Thanks
IanB

P.S. I am going to have to talk to Wilbert about being allowed to swear in this forum, I really to right now. Don't all programmers speak fluent profanity. :D