x264 Windows x64 support

[Blue_MiSfit]

I smell the need for some MeGUI updates soon to automate using 64 bit x264

~MiSfit

[skystrife]

Originally Posted by LoRd_MuldeR

LOL. Stupid my used wrong syntax to create the pipe and overwrote x264_x64.exe with random data

It does work now, like this:


However I get some continuous horizontal shift in the output file. I suspect the color format is wrong. Any ideas?


[EDIT]

Here is a sample of what I get:
http://www.mediafire.com/file/kcy4n2...x64-avs2yuv.7z

When piping using stdout x264 is expecting raw yuv input, and avs2yuv is outputting y4m. The detection for this is based on file extension, and stdout has none, so x264 wrongly assumes yuv input.

Try using avs2yuv -raw file.avs - | x264 --stuff.

/me curses the lack of named pipes in windows.

[burfadel]

Originally Posted by LoRd_MuldeR

Yes, passing data via STDOUT from a 32-Bit process to a 64-Bit process (or vice versa) should be possible.
That's because in that case we have to two completely separate processes. No 32-Bit and 64-Bit code is mixed in the same process.
I suspect I did not call it properly, because I didn't work with x264 and input from STDOUT so far...

I guess passing data via stdout is kind of like sending the frames to a buffer?

I wasn't sure of the exact issues, but it does appear then that it is possible to use x64 x264 using 32 bit avisynth & Gui frontends???

[LoRd_MuldeR]

Originally Posted by burfadel

I guess passing data via stdout is kind of like sending the frames to a buffer?

I pipe is nothing but a FIFO buffer. Similar to a message queue.

Originally Posted by burfadel

I wasn't sure of the exact issues, but it does appear then that it is possible to use x64 x264 using 32 bit avisynth & Gui frontends???

Should be possible indeed! The front-end needs to call like this:

cmd.exe /c ""C:\Path to Tools\avs2yuv.exe" "C:\Path to Source\input.avs" -raw - | "C:\Path to Tools\x264_x64.exe" --crf 22 --progress --output "C:\Path to Output\output.mkv" -"

[roozhou]

Originally Posted by LoRd_MuldeR

I pipe is nothing but a FIFO buffer. Similar to a message queue.



Should be possible indeed! The front-end needs to call like this:

cmd.exe /c ""C:\Path to Tools\avs2yuv.exe" "C:\Path to Source\input.avs" -raw - | "C:\Path to Tools\x264_x64.exe" --crf 22 --progress --output "C:\Path to Output\output.mkv" -"

Are there mplayer/mencoder builds for win64? It should make things easier.

[LoRd_MuldeR]

Originally Posted by skystrife

When piping using stdout x264 is expecting raw yuv input, and avs2yuv is outputting y4m. The detection for this is based on file extension, and stdout has none, so x264 wrongly assumes yuv input.

Try using avs2yuv -raw file.avs - | x264 --stuff.

That works flawlessly! Thank you!

Originally Posted by skystrife

/me curses the lack of named pipes in windows.

Windows does have named pipes. No idea how to create them from in the shell, but:
http://msdn.microsoft.com/en-us/libr...50(VS.85).aspx

[LoRd_MuldeR]

Originally Posted by roozhou

Are there mplayer/mencoder builds for win64? It should make things easier.

Not really. Even if you had a working MEncoder x64 build with x264 x64 enabled, you'd still need Avisynth x64
Unless you don't work with Avisynth and rely on MEncoder's decoders/filters instead, of course...

The good thing with the "pipe" method is that we can use good old 32-Bit Avisynth including all the famous filters

[Snowknight26]

My preliminary results show that the x64 version is 15% faster.

1080p video, ~15000kbps, essentially max settings.
x86: 1000 frames @ 2.11fps
x64: 1000 frames @ 2.44fps

C:\temp\Encoding\verticallimit>cd ..

C:\temp\Encoding>avs2yuv.exe verticallimit\verticallimit.avs -raw -   | x264.x64
 --pass 2 --bitrate 15093 --stats "verticallimit/verticallimit.stats" --bframes
3 --level 4.1 --ref 4 --mixed-refs --no-fast-pskip --b-pyramid --weightb --direc
t auto --deblock -3:-3 --subme 9 --trellis 2 --psy-rd 1.0:0 --analyse all --8x8d
ct --no-dct-decimate --me umh --threads auto --thread-input --progress --no-psnr
 --output "verticallimit/verticallimit.mkv" - 1912x1040
x264 [warning]: width or height not divisible by 16 (1912x1040), compression wil
l suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
verticallimit\verticallimit.avs: 1912x1040, 24000/1001 fps, 1000 frames
x264 [info]: profile High, level 4.1
x264 [info]: slice I:14    Avg QP:15.25  size:128229
x264 [info]: slice P:505   Avg QP:19.12  size: 89484
x264 [info]: slice B:481   Avg QP:21.82  size: 53901
x264 [info]: consecutive B-frames: 26.7% 20.1% 14.3% 38.9%
x264 [info]: mb I  I16..4: 29.0% 60.4% 10.5%
x264 [info]: mb P  I16..4:  1.8%  7.0%  0.5%  P16..4: 47.1% 24.7%  8.7%  0.1%  0
.1%    skip:10.0%
x264 [info]: mb B  I16..4:  0.0%  0.8%  0.0%  B16..8: 57.6%  1.1%  2.2%  direct:
11.2%  skip:27.1%  L0:44.2% L1:50.7% BI: 5.1%
x264 [info]: 8x8 transform  intra:73.0%  inter:45.9%
x264 [info]: direct mvs  spatial:87.3%  temporal:12.7%
x264 [info]: ref P L0  75.6% 13.4%  6.7%  4.4%
x264 [info]: ref B L0  68.2% 22.5%  9.3%
x264 [info]: ref B L1  92.2%  7.8%
x264 [info]: SSIM Mean Y:0.9486720
x264 [info]: kb/s:14582.2

encoded 1000 frames, 2.44 fps, 14582.61 kb/s

C:\temp\Encoding>ping 127.0.0.1

Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

Ping statistics for 127.0.0.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\temp\Encoding>x264 --pass 2 --bitrate 15093 --stats "verticallimit/verticalli
mit.stats" --bframes 3 --level 4.1 --ref 4 --mixed-refs --no-fast-pskip --b-pyra
mid --weightb --direct auto --deblock -3:-3 --subme 9 --trellis 2 --psy-rd 1.0:0
 --analyse all --8x8dct --no-dct-decimate --me umh --threads auto --thread-input
 --progress --no-psnr --output "verticallimit/verticallimit.mkv" "verticallimit/
verticallimit.avs"
avis [info]: 1912x1040 @ 23.98 fps (1000 frames)
x264 [warning]: width or height not divisible by 16 (1912x1040), compression wil
l suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [warning]: 2nd pass has fewer frames than 1st pass (1000 vs 179031)
x264 [info]: profile High, level 4.1
x264 [info]: slice I:14    Avg QP:15.11  size:131874
x264 [info]: slice P:505   Avg QP:18.97  size: 93347
x264 [info]: slice B:481   Avg QP:21.68  size: 56405
x264 [info]: consecutive B-frames: 26.7% 20.1% 14.3% 38.9%
x264 [info]: mb I  I16..4: 28.6% 60.5% 10.9%
x264 [info]: mb P  I16..4:  1.9%  7.7%  0.6%  P16..4: 46.0% 24.9%  8.8%  0.1%  0
.1%    skip: 9.9%
x264 [info]: mb B  I16..4:  0.0%  0.9%  0.0%  B16..8: 57.5%  1.1%  2.3%  direct:
11.5%  skip:26.6%  L0:44.0% L1:50.4% BI: 5.6%
x264 [info]: 8x8 transform  intra:74.3%  inter:45.0%
x264 [info]: direct mvs  spatial:87.3%  temporal:12.7%
x264 [info]: ref P L0  75.6% 13.3%  6.7%  4.4%
x264 [info]: ref B L0  68.5% 22.3%  9.3%
x264 [info]: ref B L1  92.2%  7.8%
x264 [info]: SSIM Mean Y:0.9500119
x264 [info]: kb/s:14599.9

encoded 1000 frames, 2.11 fps, 14600.31 kb/s

C:\temp\Encoding>pause
Press any key to continue . . .

[qyqgpower]

the output of mingw x64 version seems to be different from msvc and normal 32bit version. (of course, without --threads option)

mingw x64
x264 [info]: SSIM Mean Y:0.9804991
x264 [info]: PSNR Mean Y:44.020 U:47.821 V:48.225 Avg:44.906 Global:41.938 kb/s:1574.50
encoded 2184 frames, 19.59 fps, 1574.64 kb/s

msvc2008 x64
x264 [info]: SSIM Mean Y:0.9804984
x264 [info]: PSNR Mean Y:44.019 U:47.820 V:48.224 Avg:44.905 Global:41.938 kb/s:1574.53
encoded 2184 frames, 19.65 fps, 1574.68 kb/s

mingw x86
x264 [info]: SSIM Mean Y:0.9804984
x264 [info]: PSNR Mean Y:44.019 U:47.820 V:48.224 Avg:44.905 Global:41.938 kb/s:1574.53
encoded 2184 frames, 18.56 fps, 1574.68 kb/s

[roozhou]

Originally Posted by LoRd_MuldeR

Windows does have named pipes. No idea how to create them from in the shell, but:
http://msdn.microsoft.com/en-us/libr...50(VS.85).aspx

Rename attached file to Namedpipe.exe

usage:

Namedpipe pipename exename args

1) pipename must start with \\.\pipe\
2) exename is the program that you want to redirect its stdin
3) args is extra command-line arguments for exename, remember to put at least 2 spaces between exename and args

e.g. pipe mplayer's to neroAacEnc

1) in 1st cmd window, type "Namedpipe \\.\pipe\1.wav neroAacEnc.exe -q 0.3 -ignorelength -if - -of xxx.mp4"
2) in 2nd cmd window, type "mplayer xxx.xxx -vc NULL -vo NULL -ao fast:file=\\.\pipe\1.wav"

Source code(compiled in VC2005)

#pragma comment(linker,"/INCREMENTAL:NO /SECTION:.text,EWR /ALIGN:0x1000")
#include <Windows.h>

#define BUFSIZE 0x400000
char processname[256];
char pipename[64];
char *cmd,*tmp;
HANDLE hPipe/*,hCP,hChildIn*/;
STARTUPINFOA stinfo={0};
PROCESS_INFORMATION pinfo={0};
SECURITY_ATTRIBUTES sa={sizeof(SECURITY_ATTRIBUTES),NULL,TRUE};
void WINAPI Main()
{
	DWORD quoteon,state,cmdend;
	cmd = GetCommandLineA();
	quoteon = 0;
	state = 0;
	cmdend=0;
	tmp = pipename;
	while (!cmdend)
	{
		switch(*cmd)
		{
		case 0:
			cmdend=1;
			*tmp=0;
			goto END;
		case '\"':
			quoteon = !quoteon;
			break;
		case ' ':
		case '\t':
			if (quoteon)
			{
				switch(state)
				{
				case 2:
				case 4:
					*tmp++ = *cmd;
				}
			}
			else
			{
				switch(state)
				{
				case 0:
					state++;
					break;
				case 4:
					cmdend = 1;
				case 2:
					*tmp = 0;
					state++;
				}
			}
			break;
		default:
			switch(state)
			{
			case 3:
				tmp = processname;
			case 1:
				state++;
			case 2:
			case 4:
				*tmp++ = *cmd;
			}
		}
		cmd ++;
	}
END:
	if (state<2 || quoteon)
		return;
	if (INVALID_HANDLE_VALUE == (hPipe = CreateNamedPipeA(pipename,PIPE_ACCESS_DUPLEX,PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT,PIPE_UNLIMITED_INSTANCES,BUFSIZE,BUFSIZE,0,&sa)))
		return;
	ConnectNamedPipe(hPipe,NULL);
	stinfo.cb = sizeof(STARTUPINFO);
	stinfo.dwFlags = STARTF_USESTDHANDLES;
	stinfo.hStdInput = hPipe;//hChildIn;
	stinfo.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
	stinfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
	if (CreateProcessA(processname,cmd,0,0,TRUE,0,NULL,NULL,&stinfo,&pinfo))
	{

		WaitForSingleObject(pinfo.hProcess,INFINITE);
	}
}

[kemuri-_9]

Originally Posted by qyqgpower

the output of mingw x64 version seems to be different from msvc and normal 32bit version. (of course, without --threads option)

which mingw64 build did you use?
...asking as there's quite a few roaming around now...

[qyqgpower]

Originally Posted by kemuri-_9

which mingw64 build did you use?
...asking as there's quite a few roaming around now...

x64 builds are from #47 and #48
x86 build is from MeGUI's auto update (Skystrife's patched build)

commandline:
avs2yuv -raw "NCOP.avs" - | "x264.exe" --crf 22 --progress --sar 32:27 --output "F:\temp.mkv" - 720x480

I did another test without piping with x86 build:

x264 [info]: SSIM Mean Y:0.9804984
x264 [info]: PSNR Mean Y:44.019 U:47.820 V:48.224 Avg:44.905 Global:41.938 kb/s:1510.04
encoded 2184 frames, 19.81 fps, 1510.18 kb/s
output is exactly at same size (17,226,071 bytes) so the bitrate calculation here is more precision, but this is not very important.
The key is that the speed penalty of piping is larger than the speed advantage of x264 win64 build.

[roozhou]

Originally Posted by qyqgpower

x64 builds are from #47 and #48
x86 build is from MeGUI's auto update (Skystrife's patched build)

commandline:
avs2yuv -raw "NCOP.avs" - | "x264.exe" --crf 22 --progress --sar 32:27 --output "F:\temp.mkv" - 720x480

I did another test without piping with x86 build:

x264 [info]: SSIM Mean Y:0.9804984
x264 [info]: PSNR Mean Y:44.019 U:47.820 V:48.224 Avg:44.905 Global:41.938 kb/s:1510.04
encoded 2184 frames, 19.81 fps, 1510.18 kb/s
output is exactly at same size (17,226,071 bytes) so the bitrate calculation here is more precision, but this is not very important.
The key is that the speed penalty of piping is larger than the speed advantage of x264 win64 build.

Piping of Windows cmd.exe sucks.

[Dark Shikari]

Is the piping penalty mitigated by multithreading?

[kemuri-_9]

piping is also not required if one has avisynth x64.

but now one may say 'but none of my filters work in x64!'

you can either
1. create a lossless of your filtering from the x86 version to plainly open in avisynth x64 with AVISource()
(as there are x64 versions of lags and ffdshow)

2. pipe from avisynth x86 to x64 using the TCPDeliver plugin
(compatible x86 and x64 versions are on squid_80's avs x64 filter collection page - url is floating around somewhere on doom9...)
(also not sure if this is going to be more efficient than stdout piping or not... - my guess is it'll be worse to some degree)

3. limit what you're doing to what there is x64 filters for
(i lol inside as i write this last choice)

the options are there... though with differing degrees of feasibility...

[burfadel]

Nag Microsoft to allow high level mixed mode x86/64 instead of disallowing it altogether. Its purely a security reason why its disallowed, not a technical one By high level I mean non system level operations, such as antivirus & firewall (which require a 'driver' to allow access to the system, in which case the driver can be 64 bit and the programme 32 bit), device drivers etc that allow 'low level' access to the system. It is quite pointless disallowing mixed mode for instances of passing processed data in my opinion.

[LoRd_MuldeR]

Originally Posted by burfadel

Nag Microsoft to allow high level mixed mode x86/64 instead of disallowing it altogether. Its purely a security reason why its disallowed, not a technical one By high level I mean non system level operations, such as antivirus & firewall (which require a 'driver' to allow access to the system, in which case the driver can be 64 bit and the programme 32 bit), device drivers etc that allow 'low level' access to the system. It is quite pointless disallowing mixed mode for instances of passing processed data in my opinion.

Well, I'm not aware of any OS that allows 32-Bit and 64-Bit code to run in the context of the same process.
And it is possible to pass data between 32-Bit and 64-Bit processes, either trough pipes or through network sockets.

[squid_80]

Originally Posted by burfadel

Its purely a security reason why its disallowed, not a technical one

It's got nothing to do with security. How do you mix code that has different registers and pointer sizes?
If you want to do IPC between 32/64-bit programs the quickest method is to map the same chunk of memory into both processes and restrict access with an event/semaphore etc.

[burfadel]

Thats why I mentioned 'high level' access. It doesn't matter if there are different registers and pointer sizes for high level access, which this is. If it were a programme loading a DLL, then it would matter Even encoding using x264 and Avisynth is high level, as Avisynth merely serves the frames (that is, they're separate)!

[LoRd_MuldeR]

If an application (e.g. x264) uses Avisynth input, then it unavoidably needs to load the Avisynth DLL.
And mapping a 32-Bit DLL into the context of a 64-Bit process (or vice versa) isn't possible for the reasons mentioned by squid_80.

So you got exactly two options to use 64-Bit x264 with Avisynth input:
* Use 64-Bit Avisynth and accept the limitations it currently has.
* Call 32-Bit Avisynth from a separate 32-Bit process and transfer the "raw" data to the 64-Bit x264 process (may it be via pipe, via network socket or maybe via shared memory).

I don't know if it is possible to share memory between 32-Bit and 64-Bit processes on Windows, but pipes do work for sure.