Pipe x264 stats file to external program

[omion]

This is going to sound really weird, but is there any way to redirect the x264 stats file to an external program, or at least output it on STDOUT/STDERR? I thought that's what --verbose did, but it doesn't have all the data that's in the stats file.

I've heard things about "named pipes", but I'm on Windows so I don't think it applies.

I ask because I'm making a distributed-encoding server for x264, which will send out batches of frames to different computers across the network, then assemble them together when each batch is done. It's similar to Tobias's ELDER programs, but works over the network. (It should hopefully be a bit more efficient, too ;) )

I want each line of the first pass to be sent back to the central computer as soon as it's done, so I don't want to wait until the whole stats file is written. I suppose I could read the temp stats file as it's being written, but that requires quite a bit of nasty hackery.

The second pass will be a bit different, but I at least want to get the first pass done by sending back individual frames.

Thanks!

[morph166955]

well i see two ways to do this depending on your OS

I do all my encoding in a freebsd environment since my server has alot more power then my desktop does. if your on a windows box i assume you can execute x264 through cygwin. if you do that, you may be able to some how use a pipe in bash to throw it to a specific program. if not, you can probably use cygwin as well as "tail -f statsfile" and pipe the tail output too the program you want to use as its stdin.

[lcx]

If you want to output to stdout, just copy lines 801 to 810 in encoder\ratecontrol.c
and add another line with

fprintf( stdout, ...

Also, named pipes do work on Windows.
Use CreateNamedPipe to create the pipe; and use the WriteFile/ReadFile Win APIs to write/read data on the server/client repectively.

[hitbit]

Also, named pipes do work on Windows.
Use CreateNamedPipe to create the pipe; and use the WriteFile/ReadFile Win APIs to write/read data on the server/client repectively.
This works only in a programming environment, right? I.e., there's nothing like mkfifo on Windows' shell, or I am wrong?

[omion]

If you want to output to stdout, just copy lines 801 to 810 in encoder\ratecontrol.c
and add another line with

fprintf( stdout, ...

This might work. I was hoping to avoid changing the source, but I suppose I could do it. There are two problems I currently have with this (both related to my inability to program C:
1) One of my computers has 32-bit Windows on it, and I've never compiled x264 for 32-bits.
2) The current builds of x264 break when it comes to 64-bit Windows. It's tailored to 64-bit Linux, and the assembly is quite a bit different from Windows. I had a patch by squid_80 which fixed it for older revisions, but it doesn't work on current ones.

Also, named pipes do work on Windows.
Use CreateNamedPipe to create the pipe; and use the WriteFile/ReadFile Win APIs to write/read data on the server/client repectively.
That's interesting. I guess they do. The problem is that I'm coding in OCaml, and I do not want to write any more C <-> OCaml glue code. I don't know C well enough to write it.

I'd really like one of two things to happen:
1) support for something like "--stats-stdout" which would copy the stats to stdout (like what you mentioned with changing ratecontrol.c, but controlled by an argument. I'd also like this to be put into SVN, but that's probably too much to ask)
2) Somebody magically comes up with a working mkfifo program. I downloaded a mkfifo for Windows, but it doesn't work...

It looks like morph166955's "tail -f" idea should work, but x264 doesn't flush after writing a frame (it's generally not necessary) so I'm still getting it in chunks.

I think the problem is that I'm being too picky. I suppose there's nothing wrong with reading the stats file after it's been written. That's the easiest way to do it, too. I think I'll give up on the real-time frame sending part and just do batches at the end.

[squid_80]

Is it just pixel-a.asm that's broken?

[omion]

I think the only problem is in dct-a.asm. r479 conflicts quite heavily with your changes. pixel-a.asm seems to be OK.

[squid_80]

ftp://squid80.no-ip.com/dct-a.asm

The SVN code for x264_sub8x8_dct8_sse2 and x264_add8x8_idct8_sse2 is trashing xmm6-xmm15. I'm not sure if my method is exactly right (stupid pro/epilog restrictions) but at least it saves them.

Edit: pixel-a.asm breaks at r490 I think. Requires a bit of thinking to fix cleanly.

[omion]

Your new dct-a works great, thanks!

You're right that pixel-a breaks, though. I was just looking at the conflicts, but it looks like a bunch of stuff was added. Compiling r490 gives a whole slew of errors.

[squid_80]

Argh, the new functions in pixel-sse2.asm aren't saving the xmm registers either. Plus they call each other, which is pretty wacky. Haven't had time to look in detail.

I might have another idea for your original problem too. Try using con as the statsfile name, e.g. "--stats con". It spits the stats info directly out to the console. Whether or not you can catch it with something else I'm not sure.