x264 Windows x64 support

[burfadel]

Ah ok! I was forgetting about the Avisynth DLL, in the context for which its loaded!

[squid_80]

3. limit what you're doing to what there is x64 filters for
(i lol inside as i write this last choice)Hey it works for me. I stopped doing x64 ports of avisynth plugins because I had everything I commonly use and everytime I ported a plugin at someone else's request they wouldn't bother giving me feedback on whether it even worked or not.

There was a brief attempt at creating a fourth option: Use avisynth x64 and use a filter32() plugin to use 32-bit filters. This worked by spinning off a 32-bit child avisynth process and using shared memory for IPC. It didn't get far off the ground though because I'm primarily a codemonkey and kinda suck at analysis and design.
Even encoding using x264 and Avisynth is high level, as Avisynth merely serves the frames (that is, they're separate)! AVIStreamRead is definitely not what I'd call a high-level call. For a start the function call uses four pointers. To go high level you'd have to use COM objects with custom marshalling code. Down that route lies madness.
I don't know if it is possible to share memory between 32-Bit and 64-Bit processes on Windows, but pipes do work for sure.Definitely possible. The only tricky part is you either use a static name for the shared memory object (bad because the name is visible across the whole OS) or use a temporary name and end up with a chicken and egg scenario; the child process needs the object name to do IPC, but the host process needs IPC to tell the child process what the object name is.

[imk]

If you want to compile with ICC instead of MSVC.

I put the files together here:
http://imk.cx/pc/x264/x264_icc_windows.tar.lzma

It's pretty straight forward.

Here's three different benchmark results:

http://pastie.org/346781
http://pastie.org/346664
http://pastie.org/346782

I ran the commands as many times as it took to get consistent results and then I took the highest score. Overall there was a standard deviation of +/- 0.2 FPS.

[mariush]

A few days ago I thought it would be fun to test if x264 would run in an emulated OS so I installed the latest version of Virtualbox on my Windows 2003 R2 32bit, and installed Windows 2003 R2 64 bit on the emulator.
This x264 64 bit version passed all asm tests just fine but I didn't bother copying video in the OS image and trying to compress stuff... My guess it would work about as fast as in 32 bit, as Virtualbox supports virtualization and my Q6600 also has is.

[LoRd_MuldeR]

A few days ago I thought it would be fun to test if x264 would run in an emulated OS so I installed the latest version of Virtualbox on my Windows 2003 R2 32bit, and installed Windows 2003 R2 64 bit on the emulator.
This x264 64 bit version passed all asm tests just fine but I didn't bother copying video in the OS image and trying to compress stuff... My guess it would work about as fast as in 32 bit, as Virtualbox supports virtualization and my Q6600 also has is.

When you run a 64-Bit guest system on a 32-Bit host system, which is possible with the latest version of VirtualBox, how does it work? Since the host OS is 32-Bit, the CPU is running in "Legacy Mode". That doesn't allow VirtualBox to switch to "64-Bit Mode", even if it uses its system driver to execute in "Ring 0" of the (32-Bit) host OS. Hence VirtualBox can't allow the guest OS to execute any instructions directly on the hardware. Consequently I have to assume that VirtualBox will emulate the CPU for the 64-Bit guest system entirely in software! If that is the case, it would totally destroy your encoding speed...

[mariush]

It runs just as fast as the normal Windows 2003 32 bit OS. I couldn't spot any difference and the processor usage was basically about 1%. Quite different comparing to running Windows 2003 32 bit inside Microsoft's VirtualPC.

If you have some specific tests/benchmarks you'd like me to run, feel free to suggest and if I have time I'll run them.

ps. Here's how it's reported by CPU-Z:

http://savedonthe.net/image/165/Clipboard02.png http://savedonthe.net/image/166/Clipboard03.png

So apparently the processor is not detected exactly as the processor is configured on my system (all stock settings), but otherwise seems to work just fine.

[LoRd_MuldeR]

According to the VBox manual:

If you want to use 64-bit guest support on a 32-bit host operating system, you must also select a 64-bit operating system for the particular VM. Since supporting 64 bits on 32-bit hosts incurs additional overhead, VirtualBox only enables this support upon explicit request.

Well, "additional overhead" is not very precise. You may run an encode in your virtual 64-Bit system and encode the same file with identical settings in your native 32-Bit system. Then compare the FPS values.

Note: You don't need to copy any files over to your guest OS, because VBox supports shared folders. That is: You can access the host filesystem from within the guest OS :)


[EDIT]

VBox still requires a 64-Bit CPU with VT-x or AMD-V support to run a 64-Bit guest on a 32-Bit host OS. So it seems VT-x/AMD-V allows the VM to run 64-Bit code natively, even if the host OS is 32-Bit.

But running a 64-Bit guest OS without 64-Bit hardware is not possible. Except for pure software emulation, of course. But that would be sloooooow...

[mariush]

I'm not installing any "64-bit guest support" so I don't think that paragraph applies. Maybe if I were to use shared folders or something like that. Usually I just set a FTP server and transfer the files I need inside the emulated os-es.
I'll try to do a test encoding soon, not right now as it's 6.14 AM here and I haven't slept at all.

[LoRd_MuldeR]

I'm not installing any "64-bit guest support" so I don't think that paragraph applies.

Your host OS is Windows 2003 R2 32-Bit and you installed Windows 2003 R2 64-Bit in the VirtualBox. Hence you are definitely using the "64-bit guest support" of VBox ;)

Maybe you mixed that up with "Guest Additions". These are required for Shared Folder support and stuff, but the cited paragraph isn't talking about them...

(BTW: The guest system isn't "emulated" but "virtualized". Otherwise you'd never get an acceptable performance. VBox only emulates in software when it can't be avoided)

I'll try to do a test encoding soon, not right now as it's 6.14 AM here and I haven't slept at all.

5:19 AM here :D

[mariush]

I've managed to do a test encoding on the computer and inside the Virtualbox emulated Windows 2003 64bit..

The sample was a bit over a minute, from Heroes, 1280x720.
The parameters were:

C:\temp\x264win64_mingw>x264.exe --bitrate 4096 --level 4.1 --ref 4 --mixed-refs
--bframes 3 --b-adapt 2 --weightb --direct auto --deblock -1:-1 --subme 7 --tre
llis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 50000 --vbv-maxra
te 50000 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim -
-progress --output C:\encode.h264 c:\sample.yuv 1280x720


Here's how it works in 64bit:


x264 [info]: using cpu capabilities: MMX2 SSE2 SSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: slice I:12 Avg QP:13.24 size: 58848
x264 [info]: slice P:585 Avg QP:14.97 size: 31822
x264 [info]: slice B:929 Avg QP:16.93 size: 13128
x264 [info]: consecutive B-frames: 6.3% 20.3% 46.2% 27.2%
x264 [info]: mb I I16..4: 10.8% 75.0% 14.2%
x264 [info]: mb P I16..4: 7.8% 28.7% 3.0% P16..4: 39.3% 13.5% 5.6% 0.0% 0.0% skip: 2.1%
x264 [info]: mb B I16..4: 1.4% 1.1% 0.1% B16..8: 38.7% 0.7% 1.1% direct:25.4% skip:31.5% L0:44.5% L1:51.0% BI: 4.5%
x264 [info]: final ratefactor: 21.50
x264 [info]: 8x8 transform intra:70.1% inter:66.4%
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: ref P L0 76.0% 13.2% 7.0% 3.8%
x264 [info]: ref B L0 88.3% 7.9% 3.7%
x264 [info]: kb/s:4130.8

encoded 1526 frames, 1.66 fps, 4130.96 kb/s


What I found interesting was that Virtualbox used only 2 cores of the Q6600, and these two cores were at a pretty steady 25% usage. The other two cores were unused, running at about 1-4%.
If it were to use the full four cores and at a higher load, it probably would get closer to the results below.

Here's the normal run on the Windows 2003 R2 32bit:


x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: slice I:11 Avg QP:13.33 size: 59475
x264 [info]: slice P:561 Avg QP:14.91 size: 32500
x264 [info]: slice B:954 Avg QP:16.88 size: 13317
x264 [info]: consecutive B-frames: 4.4% 18.9% 47.9% 28.8%
x264 [info]: mb I I16..4: 9.6% 76.9% 13.5%
x264 [info]: mb P I16..4: 8.0% 29.9% 3.3% P16..4: 38.3% 13.2% 5.4% 0.0% 0.0% skip: 2.0%
x264 [info]: mb B I16..4: 1.4% 1.2% 0.1% B16..8: 38.7% 0.7% 1.1% direct:25.5% skip:31.3% L0:44.4% L1:51.0% BI: 4.6%
x264 [info]: final ratefactor: 21.45
x264 [info]: 8x8 transform intra:70.1% inter:66.3%
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: ref P L0 76.0% 13.2% 7.0% 3.8%
x264 [info]: ref B L0 88.3% 8.1% 3.6%
x264 [info]: kb/s:4140.4

encoded 1526 frames, 7.52 fps, 4140.54 kb/s


I've installed guest additions in Virtualbox and I'll restart the image and run another test later with the additions active. Will probably edit this post and paste that result here.

ps, later edit:

rebooted virtual os, now with guest additions enabled, increased memory size to 1536MB just for kicks, tried changing the ide controller to sata in case virtual drive speed was limited by the ide controller but windows blue screened at boot.

I aborted the test at about 48%, because I didn't see much improvement:


x264 [info]: using cpu capabilities: MMX2 SSE2 SSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: slice I:4 Avg QP:13.08 size: 59964
x264 [info]: slice P:266 Avg QP:15.02 size: 31061
x264 [info]: slice B:372 Avg QP:16.92 size: 12741
x264 [info]: consecutive B-frames: 8.9% 27.3% 38.1% 25.7%
x264 [info]: mb I I16..4: 10.2% 73.7% 16.2%
x264 [info]: mb P I16..4: 7.1% 26.3% 2.9% P16..4: 41.2% 14.2% 5.8% 0.0% 0
.0% skip: 2.5%
x264 [info]: mb B I16..4: 1.2% 1.0% 0.1% B16..8: 39.6% 0.7% 1.0% direct:
24.0% skip:32.4% L0:45.3% L1:50.4% BI: 4.3%
x264 [info]: final ratefactor: 21.50
x264 [info]: 8x8 transform intra:70.3% inter:66.1%
x264 [info]: direct mvs spatial:99.7% temporal:0.3%
x264 [info]: ref P L0 75.6% 13.4% 7.1% 4.0%
x264 [info]: ref B L0 88.6% 7.8% 3.6%
x264 [info]: kb/s:4125.1

aborted at input frame 642
encoded 642 frames, 1.63 fps, 4125.36 kb/s



Again, the processor is not used as it should be, but it definitely uses more than one physical core (the CPU inside the image is detected as one core, one thread Core 2 Duo by CPU-Z) :

http://savedonthe.net/image/167/Clipboard03.png

[LoRd_MuldeR]

What I found interesting was that Virtualbox used only 2 cores of the Q6600, and these two cores were at a pretty steady 25% usage. The other two cores were unused, running at about 1-4%.

I think the virtual machine only exposes one single CPU to the guest OS...

[imk]

Yes. VirtualBox only exposes one CPU to the OS. However, VirtualBox itself is multi-threaded, so there will be speedups versus just a single CPU.

[MasterNobody]

Can somebody run this checkasm (http://stashbox.org/359637/checkasm.exe) and say me at which stage it crash (or may be it run correctly but this is highly doubtful). I currently can't test it myself because I am on 32-bit PC.

[Snowknight26]

Microsoft Windows [Version 6.0.6001]
Copyright (c) 2006 Microsoft Corporation. All rights reserved.

C:\temp>checkasm.exe
x264: using random seed 1123929904
x264: MMX
- pixel sad : [OK]
- pixel sad_aligned : [OK]
- pixel ssd : [OK]
- pixel satd : [OK]

C:\temp>

[dj_tjerk]

It just prints the first two lines here, no results whatsoever :| (but that might be my system... Win2003 x64 on an ol' X2 4400)

[LoRd_MuldeR]

It just prints the first two lines here, no results whatsoever :| (but that might be my system... Win2003 x64 on an ol' X2 4400)

Same here:

C:\Downloads>checkasm
x264: using random seed 1962273904
x264: MMX

C:\Downloads>

[MasterNobody]

Thanks. Then I will debug it tomorrow on 64-bit PC and upload new version with new patch.

[MasterNobody]

OK. Here is new checkasm: http://stashbox.org/360692/checkasm.zip. It contains 3 versions: 1 - check only results (and support large addresses), 2 - check result and nonvolatile registers (don't support large addresses), 3 - check result and nonvolatile registers (and support large addresses but will crash if they really used). So 1 and 2 must run fully correct and 3 can crash on the start if addresses large than 32 bit used. I am particularly interested in Core2, Core i7 and Phenom results (of all checkasm versions) so specify yours CPU type in the posts.

[vucloutr]

Q6600: "x264: All tests passed Yeah :)" for all three executables.

funny thing is it does even say [OK] for PHADD which is not supported by the Q6600 (right?)

[Dark Shikari]

Q6600: "x264: All tests passed Yeah :)" for all three executables.

funny thing is it does even say [OK] for PHADD which is not supported by the Q6600 (right?)It's supported, just not used, because it's slower on pre-Penryn CPUs.

[kemuri-_9]

Phenom here
CPUFlags = MMX2 SSE2Fast SSEMisalign LZCNT

checkasm1:
no errors

checkasm2 & 3:

x264: SSE_Misalign
- pixel sad_x3 : [OK]
- pixel sad_x4 : [OK]
- mc luma : [OK]
failed to preserve register
- hpel filter : [FAILED]

[RickA]

Greets,

Q6600 here and all tests passed.

Glad there is activity on this thread again. I'm really excited to see some work done towards a 64bit x264 for Windows. So much so that I am trying to learn to compile Handrake and x264 via the instructions here http://trac.handbrake.fr/wiki/CompileOnWindows . Have the suggested software installed. Just need to figure out where to go from there.

Unfortunetly, my searches for guides have turned up more info on Linux compiling and less info on 32bit to 64bit while staying within the Windows realm. If anyone has some good suggestive reading for someone new to compiling code I'd welcome it. Willing to help the cause in whatever capacity I can.

Cheers,
Rick

[MasterNobody]

Phenom here
CPUFlags = MMX2 SSE2Fast SSEMisalign LZCNT

checkasm1:
no errors

checkasm2 & 3:

x264: SSE_Misalign
- pixel sad_x3 : [OK]
- pixel sad_x4 : [OK]
- mc luma : [OK]
failed to preserve register
- hpel filter : [FAILED]


Try with updated version: http://stashbox.org/360826/checkasm.zip

Also it is interesting to test with this registry hack (need reboot):
VirtualAlloc usually returns low addresses before high addresses. Therefore, your process may not use very high addresses unless it allocates a lot of memory or has a fragmented virtual address space. To force allocations to allocate from higher addresses before lower addresses for testing purposes, specify MEM_TOP_DOWN when calling VirtualAlloc or set the following registry value to 0x100000:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference
checkasm1&2 must work correctly and checkasm3 must crash with this hack.

[kemuri-_9]

Try with updated version: http://stashbox.org/360826/checkasm.zip

that set has no errors for me.

[MasterNobody]

If there are no other error reports than here is new patch: x264_win64_support.02.r1077.diff (http://stashbox.org/361189/x264_win64_support.02.r1077.diff)

kemuri-_9
Did you try to test with registry hack?

[kemuri-_9]

Did you try to test with registry hack?

no... i hate rebooting the computer
and you shouldn't have people hack the registry for it to work anyway.

[MasterNobody]

and you shouldn't have people hack the registry for it to work anyway.
This is not for its working this is only for testing of large address support (addresses which can't fit in 32bit).

[kemuri-_9]

a possibility for testing is by using large mallocs and increment into it for what you actually needed for work/manipulation.

#define ADDR_32_MAX 2 * 1024 * 1024 * 1024
char *baseptr, *ptr;
baseptr = malloc( ADDR_32_MAX + i_size );
if( !baseptr )
{
printf( "large malloc failed (no resources)\n" );
exit(1);
}
ptr = baseptr + ADDR_32_MAX;

....

free( baseptr );

something like that.

[squid_80]

You'd rather make all mallocs attempt to grab >2GB (which wouldn't really be sufficient, it would need to be >4GB) rather than just flipping a registry flag?

[imk]

If there are no other error reports than here is new patch: x264_win64_support.02.r1077.diff (http://stashbox.org/361189/x264_win64_support.02.r1077.diff)

First, the old patch still works for me. However, since this is newer, I'm trying to make a build using this one instead.

How should this be compiled now? I noticed you added definition checks for WIN64 rather than ARCH_X86_64.

Previously, when compiling the objects, I would specify /DARCH_X86_64. When compiling x264 itself, I would specify /DWIN32 /DARCH_X86_64. The flags I used with yasm were -DARCH_X86_64 -D_WIN64.

Using these flags now will result in yasm spitting out errors.

If I remove -DARCH_X86_64 and change -D_WIN64 to -DWIN64, then yasm will error out with "invalid size for operand #." However, if I keep -DARCH_X86_64 and change -D_WIN64 to -DWIN64 (so the line is -DARCH_X86_64 -DWIN64) then it will work fine. I'm not sure if this is the right way to do it since you do specific checks for WIN64, else ARCH_X86_64.

[kemuri-_9]

it should be -DWIN64 -DARCH_X86_64

technically it shouldn't need -DWIN64 since

%ifdef ARCH_X86_64
%ifidn __OUTPUT_FORMAT__,win64
%define WIN64
%endif
%endif


is there to automatically add the effect of -DWIN64 when -f win64 -DARCH_X86_64 is met

the checks he has on several occasions are

ifdef WIN64
....
elifdef ARCH_X86_64
...


which actually works, since it hits the WIN64 block only when both are defined, but it kills readability imo, so i can understand where you're getting confused.

they should be switched to

ifdef ARCH_X86_64
ifdef WIN64
...
else
...
endif
...


for easier clarity imo

[sshoecraft]

Excellent job, MasterNobody!

Now can we get ya to look at the assembler code in mplayer/mencoder? :D

I was able to compile a win64 version of x264 with GPAC (mp4) & pthreads support.

I then compiled a win64 version of mencoder/mplayer - with x264/faac/pthreads - but without the filter/swscale/liba52 asm code...

EDIT: looks like mencoder is crashing in x264_encoder_close - still debugging :/

[imk]

If you want to compile with ICC instead of MSVC.

I put the files together here:
http://imk.cx/pc/x264/x264_icc_windows.tar.lzma

It's pretty straight forward.

I updated this with the latest Win64 patch. I also got rid of all the .cmd files and created a single Makefile for compilation.

I've been maintaining more benchmarks here:
http://spreadsheets.google.com/pub?key=pbffjdC6iUPWs2HtYHwZ2VQ&hl=sv

[MasterNobody]

I need somebody with Core2 / Core i7 / Phenom CPU to run this version of checkasm (both 32-bit and 64-bit):
checkasm.zip (http://stashbox.org/378305/checkasm.zip)
Used patch:
x264_win64_support.05.r1090.diff (http://stashbox.org/378306/x264_win64_support.05.r1090.diff)

[Snowknight26]

All pass on both on my Q6600.

[Audionut]

Q6600

Windows 7 64 all ok
Windows XP 32 in a VM all ok.

[kemuri-_9]

all ok on my Phenom

[darkip]

All ok on my Core2Duo Extreme x7900

[tph]

All ok on C2D E6400 @ Vista64.

[LoRd_MuldeR]

All okay. 32-Bit and 64-Bit binary. Tested on Q6600, WindowsXP x64-Edition.

[avivahl]

I need somebody with Core2 / Core i7 / Phenom CPU to run this version of checkasm (both 32-bit and 64-bit):
checkasm.zip (http://stashbox.org/378305/checkasm.zip)
Used patch:
x264_win64_support.05.r1090.diff (http://stashbox.org/378306/x264_win64_support.05.r1090.diff)Both pass w/ ALL OK.
Vista x64+SP1. Core 2 Duo E6750.

[kemuri-_9]

x264_win64_support.06.r1093.diff

this one's checkasm works fine on my phenom in both win x64 and ubuntu x64 (vmware) (checked x86 and x64 builds in each)

[DarkZell666]

All ok (checkasm32 & 64) on Q9300 inside Vista x64+SP1 inside VMWare running on Windows XP SP3 32-bit host ;)
(Edit: using "Intel-VTx or AMD-V" execution mode for the guest of course).

[prettyboy85712]

Both pass w/ ALL OK.
Vista x64+SP1 Core i7 920

[imk]

Intel Core 2 Duo E6600 / Vista x64
Both tests passed.


However, using both x264_win64_support.05.r1090.diff or x264_win64_support.06.r1093.diff will cause x264 crash when compiling with ICC. x264_win64_support.02.r1077.diff still works fine, however.

[darkip]

However, using both x264_win64_support.05.r1090.diff or x264_win64_support.06.r1093.diff will cause x264 crash when compiling with ICC. x264_win64_support.02.r1077.diff still works fine, however.
I also found the same result when compiling using ICC

[MasterNobody]

imk, darkip
May be you would provide more information than simple "x264 crash when compiling with ICC"? I don't have ICC so I can't test it myself. Provide at least exception code, disassembled code near crashed instruction (with mark on crashed instruction), registers state, CPU type, OS type.
Also how do you compile it? YASM now needs "-f win32 -m amd64 -DARCH_X86_64" and may be "-DPREFIX" params.

[imk]

I was not aware of the need for -f win32 now. I've been using -f x64 (same as -f win64). I will try with that first and post here again.

[kemuri-_9]

it changed from -f win64 to -f win32 at akupenguin's request...

Yes, `yasm -f win32` is the same as -f win64, and this way is more convenient for ffmpeg's configure, I think.

[imk]

Yeah, I just compiled a build using -f win32 and it works fine now. Thanks. :)