x264 Windows x64 support

[skystrife]

I'm getting something odd occurring with my builds that I'm having some trouble pinpointing. The build compiles just fine and runs without crashing, but if I use my three sample videos that I use for fprofiling generally, the encoded videos of the compiled build end up being corrupt (I can upload a sample if needed). However, if I fprofile with just one video, the resulting encode from that binary is fine.

My three clips are a 200-frame sample of Elephants Dream, a 200-frame sample of Dark Shikari's LosslessTouhou.mkv, and a 100-frame sample of crowdrun.

tl;dr:


Zero Video Test = fine


One Video Test

elephants_dream = fine
LosslessTouhou = fine
crowd = fine


Two Video Test

elephants_dream & touhou = fine
touhou & crowd = fine
elephants_dream & crowd = fine

Testing elephants_dream + crowd atm, but I find it bizarre that fprofiling with the three videos would cause such an issue to occur, while it doesn't happen with just one video. Does anyone have an idea of what may be happening? (sources are are in y4m format)

(I am using the latest patch from the mailing list, but this problem is replicated using the r1077 patch as well.)

EDIT: the problem only occurs with all three videos. o_O

[Dark Shikari]

I could imagine that a slightly different fprofile would result in a slightly different compilation, which could in theory be the difference between a correct compilation and a miscompilation.

[kemuri-_9]

which mingw 64 binary build are you using?

periodically about once a week i try the latest builds and they've continually been broken (miscompile x264 at the least)

not to mention that mingw64 does use gcc 4.4.0 which has yet to release a stable version.

[skystrife]

gcc version 4.4.0 20081123 (experimental) (GCC)

I tried upgrading to the latest snapshot after the first few tries with all three vids failed, but the latest one fails on ./configure. -_-;

[komisar]

skystrife, try using a different compiler eg "gcc version 4.3.4 (prerelease)". I have no problem with him.

[skystrife]

Do you by chance have a link to a 64-bit compilation compatible gcc 4.3.4 for Windows? I can only seem to find the 4.4.0 series that the mingw-w64 project provides.

[komisar]

skystrife, i have build all mingw-w64 by hand from svn.
svn link for gcc: svn://gcc.gnu.org/svn/gcc/branches/gcc-4_3-branch

P.S. If you interest, I archive my cross 32 and 64 directory of mingw64 with gcc 4.3.4 for you...

[darkip]

It seems that latest recent revision of x264 doesn't work with x264_win64_support.06.r1093.diff, I think they've played with some of the assembly

[kemuri-_9]

It seems that latest recent revision of x264 doesn't work with x264_win64_support.06.r1093.diff, I think they've played with some of the assembly

Dark_Shikari added in some new assembly:
x264_add16x16_idct_dc_[mmx,sse2,ssse3]

and altered the quant asm functions to have a return type of int rather than void while adding in versions that see benefit on sse4 processors only (so thus called sse4 versions).

[Dark Shikari]

I have reports that MasterNobody's x64 patch is causing serious video corruption issues with some settings, so I would advise against using it until they are fixed.

[MasterNobody]

I have reports that MasterNobody's x64 patch is causing serious video corruption issues with some settings, so I would advise against using it until they are fixed.
May be you will report this to me, so I can reproduce it (also which version of patch were used? And which build was it?).

[Dark Shikari]

May be you will report this to me, so I can reproduce it (also which version of patch were used? And which build was it?).

http://www.stfcc.org/misc/x264/bug

[MasterNobody]

http://www.stfcc.org/misc/x264/bug
Version of patch? 32-bit or 64-bit build of x264? fprofiled build (if yes than what about without profiling because gcc 4.4.0 is very unstable at profiling)?

As I guess it was buggy (due fprofiling) skystrife's build: x264.1093M.x64.exe (http://skystrife.com/x264/x264.1093M.x64.exe)

[MasterNobody]

By the way:
New version of patch compatible with revision 1096:
x264_win64_support.07.r1096.diff (http://stashbox.org/383993/x264_win64_support.07.r1096.diff)
and binaries of checkasm for testing:
checkasm_1096.zip (http://stashbox.org/384000/checkasm_1096.zip)

[komisar]

IMHO this is the gcc-bug. I test Dark Shikari sequence and don't see this bug.
x264.1093kMod.k8.x64.exe (http://komisar.gin.by/x264.1093kMod.k8.x64.exe) fprofiled with x264_win64_support.06.r1093.diff
x264.1093kMod.k8.x64.exe --version
x264 0.66.1093kMod 1df50b9
built on Jan 29 2009, gcc: 4.3.4 20090128 (prerelease) (by_Komisar)

x264.1096M.x64.kom.exe (http://komisar.gin.by/test/x264.1096M.x64.kom.exe) with x264_win64_support.07.r1096.diff
x264.1096M.x64.kom.exe --version
x264 0.66.1096M 4c171c3
built on Feb 1 2009, gcc: 4.3.4 20090128 (prerelease) (by_Komisar)

x264.1096M.x64.kom.profiled.exe (http://komisar.gin.by/test/x264.1096M.x64.kom.profiled.exe) fprofiled with x264_win64_support.07.r1096.diff

[Dark Shikari]

If there's a bug in GCC, you should try to find exactly what code is being miscompiled. (I've found in the past it was possible to massage the code into making GCC compile it correctly.)

[komisar]

As you may remember, I tried to use gcc-4.4 earlier and pointed out issues with fprofile. But while gcc-4.4 very unstable I refused to use it.

[kemuri-_9]

komisar, could you make a x86_64 capable binary set of gcc 4.3.3 since it's officially out now?

cross-compiler from i686 or native x86_64 bins, whatever works for you/whatever you've been doing.

or even list the steps you've been taking to compile the x86_64 versions so we can make our own.

Edit:
skystrife, in the meantime have you checked to see if the x86_64 bin set 20090110 in mingw64's build list has the same problems?

[komisar]

kemuri-_9, ok. This my toolchain for crosscompile x86_64-pc-mingw32:
http://komisar.gin.by/test/mingw.komisar.7z
Step for make you own toolchain described in "mingw-w64/mingw-w64-doc/howto-build/mingw-w64-howto-build.txt" from
svn https://mingw-w64.svn.sourceforge.net/svnroot/mingw-w64/trunk

P.S. for correct compiling x264 with my mingw need patch of Makefile:libx264.a: .depend $(OBJS) $(OBJASM)
$(AR) rc libx264.a $(OBJS) $(OBJASM)
$(RANLIB) libx264.a
and%.o: %.asm
$(AS) $(ASFLAGS) -o $@ $<
# delete local/anonymous symbols, so they don't show up in oprofile
-@ $(STRIP) -x $@
and after configure change config.makARCH=X86_64
SYS=MINGW
AR=x86_64-pc-mingw32-ar
RANLIB=x86_64-pc-mingw32-ranlib
STRIP=x86_64-pc-mingw32-strip
CC=x86_64-pc-mingw32-gcc
P.P.S. Dark Shikari, gcc version 4.4 is unusable with fprofile. His make wrong .gcda-files for second pass of make with fprofiled.

[Snowknight26]

As I guess it was buggy (due fprofiling) skystrife's build: x264.1093M.x64.exe (http://skystrife.com/x264/x264.1093M.x64.exe)

Good guess. That was the build being used.

Just tested both of komisar's r1096 builds and it does not seem to happen.

[MasterNobody]

By the way I am waiting for test results of checkasm on Core2 / Core i7 / Phenom CPUs: post1244319 (http://forum.doom9.org/showthread.php?p=1244319#post1244319)

[darkip]

By the way I am waiting for test results of checkasm on Core2 / Core i7 / Phenom CPUs: post1244319 (http://forum.doom9.org/showthread.php?p=1244319#post1244319)

All tests pass on my Core2Duo Exteme x7900

[RickA]

All tests passed on my Q6600 for checkasm_1096 both 32 and 64. Really appreciate the work you all are doing on this. :-)

Cheers,
Rick

[DarkZell666]

Checkasm_1096 all ok (32 and 64) on Q9300 (Vista x64).

[paulvdb]

Checkasm_1096 all ok on i7 (XP x64).

[Esurnir]

Checkasm_1096_32/64 all ok on Memrom Core 2 Duo T5800 Vista 64 bit

[LoRd_MuldeR]

Checkasm_1096 all OK on Q6600, 32-Bit and 64-Bit, WindowsXP x64-Edition

[kemuri-_9]

oh right... the checkasms are fine on my phenom.

[TEB]

checkasm is fine here vista64 sp1 + Q9550

[Buggle]

By the way:
New version of patch compatible with revision 1096:
x264_win64_support.07.r1096.diff (http://stashbox.org/383993/x264_win64_support.07.r1096.diff)
and binaries of checkasm for testing:
checkasm_1096.zip (http://stashbox.org/384000/checkasm_1096.zip)
Both checks are fully OK
System: Merom T5500 on Windows 7 x64

PS. I run the checks in a normal cmd instance, or is administrative access necessary for this check?

[Buggle]

As I do not understand 90% of what is going around here, I was hoping if it were possible for someone to create some sort of a x264 x64 site somewhere, maybe even on the doom9 servers or on x264.nl. Such a site would track the developments that surround the x64 edition of x264, give some heads up of where to start (including the possibilities of 64bit encoding and what is needed for it) and give users like me, who have no clue about programming, compiling or patches or anything, a heads-up of where they can help to speed up development. Or just make this one a sticky and update the first post to reflect 'things that are done and/or things to be done'. I would personally prefer the latter option.

The reason for this is because a speed up of around 10% is quite a large amount in the land of encoding to my eyes, and x64 is becoming more and more 'regular' nowadays with computers having 4+GB, especially with Vistas high memory requirements and memory prices this low (I bought Kingston Hyper-X 2GB DDR2 CAS5 for 25 euros the other day). I find it surprising that so little is done (at least, that is my perception) to fully support x64 from the ground up. In a year or five it should be the main focus at the expense of x32, in my eyes.

Please don't take this the wrong way, of course I am glad that at least are people putting effort in developing this in the first place and I really appreciate that. And if I could, I would help. I just think that this topic requires and deserves more attention and coordination. Which way you turn it, it is still the future.

[kemuri-_9]

x64 has been supported for a long time, just not on windows since it has its (guess what) own standard for it.

if you were on a linux system, you'd have no problems with x64 support at all.
it's windows that's throwing the kink into the compatability and with the main devs not using windows,
it takes a windows user that knows what's going on (like MasterNobody) to work on the support for it.

[Buggle]

x64 has been supported for a long time, just not on windows since it has its (guess what) own standard for it.

if you were on a linux system, you'd have no problems with x64 support at all.
it's windows that's throwing the kink into the compatability and with the main devs not using windows,
it takes a windows user that knows what's going on (like MasterNobody) to work on the support for it.

Hmm, I didn't know that. As I said, I know nothing about programming and its kinks (of course I do know MS creates its own standards for just about everything) so I thought having platform independent code means it can automatically run on windows. Guess it doesn't.

But with the (I presume) majority part of the user base running windows my previous post hasn't lost it's value, I guess. It would still be nice to make this a sticky and adjust the opening post a little bit to reflect what is going on in this thread.

[laserfan]

...if you were on a linux system, you'd have no problems with x64 support at all.Is there a thread or site somewhere that 64-bit encoding using x264 is discussed?

Might be fun to dual-boot to a Linux platform if it's not TOO difficult, or require tons of "tribal knowledge".

[kemuri-_9]

Is there a thread or site somewhere that 64-bit encoding using x264 is discussed?

Might be fun to dual-boot to a Linux platform if it's not TOO difficult, or require tons of "tribal knowledge".

can always join #x264 on irc.freenode.net for discussion about x264.

as for the dual-booting part, most windows users i know usually find ubuntu flavors the easiest to deal with.

anyways, i've gotten x264 to build with komisar's gcc bin set (solving its quirks to suit my needs).
ended up using ./configure --host=x86_64-pc-mingw32 --extra-ldflags=-lavifil32 (outside of my pthread and gpac dependencies) to get it to compile properly.

skystrife:
have you tried komisar's gcc 4.3.4 (pre) and see if the problems still persist?

[skystrife]

Compiled my own gcc 4.3.4 (as well as the rest of the mingw-w64 package) and all is well now. Stupid broken 4.4.0... why would they (the mingw-w64 team) even be distributing it if it's so experimental?

[LoRd_MuldeR]

Stupid broken 4.4.0... why would they (the mingw-w64 team) even be distributing it if it's so experimental?

Well, experimental means that bugs are to be expected. If nobody tries to use it on a "real" project, nobody will encounter and report the remaining bugs...

[Ranguvar]

I thought having platform independent code means it can automatically run on windows. Guess it doesn't.

The C code in x264 is indeed platform independent - mostly, anyways. But the reason x264 is so ridiculously fast is because a lot of the code is handwritten assembly - the one-step-away-from-the-machine language which C is compiled (converted) into. Since the compiler is not magic, handwritten assembly is a lot of times faster. The downside is that assembly is different for every CPU architecture. Today, that's not a huge deal, since x86 and x86-64 pretty much dominate (if you don't have one of those, either you're using an old Mac or you don't need to read this, chances are). But, you run into good old MS making up their own 64-bit standard, and there's problems. You _can_ disable the handwritten assembly and use compiled C for everything - that will work 64-bit on 64-bit Windows. But the performance loss for losing the assembly will far excede the 64-bit boost.

[squid_80]

I find it surprising that so little is done (at least, that is my perception) to fully support x64 from the ground up. In a year or five it should be the main focus at the expense of x32, in my eyes.
I did a lot of working porting avisynth, avisynth plugins and various codecs to x64: http://members.optusnet.com.au/squid_80/

Judging from the feedback I got, no-one really cares. Certainly no-one ever offered to help port avisynth plugins, which is the major thing holding back avisynth64 (Also belligerent plugin authors who refuse to release their sources even when confidentiality is promised).

[Buggle]

The C code in x264 is indeed platform independent - mostly, anyways. But the reason x264 is so ridiculously fast is because a lot of the code is handwritten assembly - the one-step-away-from-the-machine language which C is compiled (converted) into. Since the compiler is not magic, handwritten assembly is a lot of times faster. The downside is that assembly is different for every CPU architecture. Today, that's not a huge deal, since x86 and x86-64 pretty much dominate (if you don't have one of those, either you're using an old Mac or you don't need to read this, chances are). But, you run into good old MS making up their own 64-bit standard, and there's problems. You _can_ disable the handwritten assembly and use compiled C for everything - that will work 64-bit on 64-bit Windows. But the performance loss for losing the assembly will far excede the 64-bit boost.
Yeah, I know about the assembly codes existence, but since that is direct 'processor language' it should work independent from the OS, right? Or is on Windows the assembly code somehow not directly run on the proc but 'translated' by the windows application interface and only then passed on? I always understood assembly to be the mother of all platform independent (meaning OS) code since it directly instructs the processor. And since x64 is a hardware standard...
But obviously I'm misunderstanding something here :(

[Buggle]

I did a lot of working porting avisynth, avisynth plugins and various codecs to x64: http://members.optusnet.com.au/squid_80/

Judging from the feedback I got, no-one really cares. Certainly no-one ever offered to help port avisynth plugins, which is the major thing holding back avisynth64 (Also belligerent plugin authors who refuse to release their sources even when confidentiality is promised).
Then here I am thanking you for the work you did. I just didn't know about it, and will go about using it once I reinstalled my desktop to Vista x64, or start using it on my 7 x64 laptop :). I normally use MeGUI, but at Xvid times I used to do things a little more manual with GordianKnot (good old days...) so there shouldn't be a reason for me to not use this.

Could I have both the normal Avisynth and the 64 bit installed, or do I need to choose between them? I only use a few filters (SplineResize, Yadif, FieldDeinterlace, DeGrainMedian, TemporalSoften, Convolution3D, VagueDenoiser and some other softener standard in megui which I cannot find the name of) so it shouldn't be much of a problem to get that to work with some basic batch scripts.
What I'm just worried about is the resizing calculation, which I have never learned to do manual (used to use GK and now MeGUI). I would want to use MeGUI to set up all the stuff like the script, settings of x264, zones and calculate bitrates and then encode in x64 mode. There now is a 64 bit x264 and obviously a x64 avisynth, so that's all one would need. DGDecode can still be 32 bit, since it doesn't 'serve' any frames or anything, right?

Maybe it would help to poke Doom9 somewhat about this subject? Frontpage or even a new guide would increase profile a little, I guess. Would give your work the appreciation it deserves.

Edit: looked at the files on your site, but I wouldn't know where to begin: many files I do not know or do not recognize, since I (being the lazy windows user that I am) always just install programs :P. But I guess I have to register the avisynth dll by using the *.reg file, but does it then have all the usual 'included' avisynth filters? Maybe you can give some heads up about what all the files are.

[nm]

Yeah, I know about the assembly codes existence, but since that is direct 'processor language' it should work independent from the OS, right? Or is on Windows the assembly code somehow not directly run on the proc but 'translated' by the windows application interface and only then passed on? I always understood assembly to be the mother of all platform independent (meaning OS) code since it directly instructs the processor. And since x64 is a hardware standard...
Instructions are the same, but the function calling convention is different in Windows: http://en.wikipedia.org/wiki/X86_calling_conventions#Microsoft_x64_calling_convention

[Buggle]

Found an interesting link that illustrates you are ahead of your time, squid ;) http://www.dailytech.com/article.aspx?newsid=13983

[paulvdb]

I did a lot of working porting avisynth, avisynth plugins and various codecs to x64: http://members.optusnet.com.au/squid_80/

Judging from the feedback I got, no-one really cares. Certainly no-one ever offered to help port avisynth plugins, which is the major thing holding back avisynth64 (Also belligerent plugin authors who refuse to release their sources even when confidentiality is promised).
I do care and would like to thank you. The biggest problem I've had so far was that there were no (recent) 64 bits x264 builds. Now that there are 64 bits x264 builds available I have started using it.

[STaRGaZeR]

I did a lot of working porting avisynth, avisynth plugins and various codecs to x64: http://members.optusnet.com.au/squid_80/

Judging from the feedback I got, no-one really cares. Certainly no-one ever offered to help port avisynth plugins, which is the major thing holding back avisynth64 (Also belligerent plugin authors who refuse to release their sources even when confidentiality is promised).

At that time I consider normal that feedback, after all 64-bit OSes were rare. But now they're not. I've been using your Decomb64 for real time IVTC in ffdshow for quite some time now, thanks :)

On a side note TIVTC would be a really nice addition. Or do you recommend another filter for hybrid material as Decomb is not very good at it?

[kemuri-_9]

I did a lot of working porting avisynth, avisynth plugins and various codecs to x64: http://members.optusnet.com.au/squid_80/

Judging from the feedback I got, no-one really cares. Certainly no-one ever offered to help port avisynth plugins, which is the major thing holding back avisynth64 (Also belligerent plugin authors who refuse to release their sources even when confidentiality is promised).

i have to admit, i only use avisynth x64 when opening lossless avis and tossing them to x264,
maybe with some minor filtering like a downsize and or vsfilter.

there's just not enough filters available to realistically be able to filter within avisynth x64,
most of the filters would require major asm overhauls which is just too time consuming if done by a single person.

and realistically, i also don't see a major effort to port them as the work required is too astronomically high.
people can live with x86 filtering until the point in time when 3.0 becomes stable.

[squid_80]

3.0 has been in development longer than the avisynth64 branch, and there hasn't been a single preview build. Make your own conclusions about that.

Updating the asm isn't hard, especially when using intel's compiler since it allows inline asm. But the issue I had is that people would ask for plugins to be ported and I'd do them, then not get any feedback. So rather than waste my time I did the handful of plugins I personally use on a regular basis for standard conversions (NTSC->PAL) and that's where I stopped. Now I'm way too busy working (anyone remember my user title used to be unemployed?) to do it any more.

There doesn't seem to be many new plugins coming out these days, maybe with a couple of people helping out it would be possible to do a big push and get all the commonly used (open-source) plugins ported.

[Buggle]

What would 3.0 bring that it would be that special and solve all our problems?

I would be glad to help where I can, but as I said I do not know anything (well, almost) about programming. Plus I'm currently quite occupied with both my masters degree and work. Still, I'd be glad to squeeze in some time to help the community, since I've been a thankful visitor of Doom9 for quite some time already (actually, from the moment DivX 5.0 came out, which is probably somewhere around 2002 or so) and would like to donate some effort in return.

I think it would be best to start checking the needs of the user base: what is x264 used most often for and which needs does that address, or in other words, which filters are most beneficial for that? As almost anyone who is currently encoding in x264 will have x64 compatible hardware, it really is a chicken and egg story. But maybe we shouldn't get our hopes up too high: windows 7 isn't out yet, and that will probably be 'our big break', since it will probably boost the amount of x64 installations.

[Ranguvar]

What would 3.0 bring that it would be that special and solve all our problems?

Most of the people you see eagerly anticipating 3.0 (like me) are wanting AviSynth untied from the Winblows platform - meaning, it will use GStreamer or something else instead of DirectShow/VfW. Cross-platform is good (GStreamer supports Linux, Windows, Mac, BSD...), and no matter what anyone says, AviSynth over avs2yuv or avsproxy is but a clumsy hack (not the quality of the tools, it's inherent) that isn't really acceptable as a long-term solution.

There's a lot of other stuff that was supposed to come in 3.0, too... http://en.wikipedia.org/wiki/AviSynth#AviSynth_3.0_code_rewrite

[kemuri-_9]

BugMaster:
akupenguin was asking about the status of this in the dev channel,
to be informed that the current patch is broken with the latest asm additions
think you could get a fix so it could make GIT sometime in the near future?