A few requests for x264

[omion]

If I could make a few requests from the x264 devs:

1. Make stdin use binary mode for Windows compiles. This would solve the problem I noticed in this thread (http://forum.doom9.org/showthread.php?t=116700&highlight=premature). I see a few sites mentioning the line "setmode(fileno(stdin), O_BINARY);" but I'm not really sure if it would help (I don't know C...)

2. Allow the stats file to be redirected to stdout. I thought I could do it with "--stats -", but that just makes an actual file named "-". (I presume doing this would need to bypass the temp file creation, as making a "temp" stdout would not make much sense)

I'm asking for these as my distributed x264 encoder is just about ready to be released, and it basically just needs to be fine-tuned.
The first issue would necessitate me making my own builds with MSVS, and keeping them relatively up-to-date (I don't want to do that; that's what Sharktooth is for :p )
The second issue is not as important, and a fix for it would simply lessen the amount of re-encoded data on a premature exit.

Thanks!

[akupenguin]

2. So you don't actually care about stdout, you just want it to not truncate the log if it's killed/crashes? That can be solved explicitly with fflush(). It just so happens that stdout calls fflush() whenever it sees a newline, while other filehandles don't do it automatically.

[omion]

Well, I was thinking about when the controller program crashes. Here's what my program currently does:
The master "controller" program sends out a bunch of frames to each of the "agent" programs that are running
The agent programs start up x264, send all the frames to it, then when it finishes, they send the stats file back to the controller.
The controller adds the agent's stats to the real stats file.

If the controller is restarted, it figures out which frames still need to be processed by which frames are in the real stats file. Therefore, it is important to put the frames into the stats file as soon as possible. Currently, when the controller crashes, all the jobs being processed are lost, along with their frames. (*)
What I would like to do is have x264 output the stats immediately to the agent, so they can be added to the real stats file ASAP. That way, if the controller dies, it has as current a stats file as possible.

I had tried opening up the temp stats file while it was being created, but that made for major headaches (I don't exactly remember why; that was about 6 months ago)


(*) I suppose it is possible to have each agent collect the temp stats file and try to send it to the new controller if they determine that the connection was closed, but I'd rather not have to do that ;) [ edit: doing it this way would also require a verification that the controller was doing the same video, just so the wrong stats aren't sent ]

[miztadux]

Hello,

I got a little request for the x264 CLI, I hope this thread is appropriate...

I'd like to see an option basically like the "@optionsfile" option of mkvmerge (http://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html).

(It allows to specify the bulk of CL options in a plain text file, it's easier to use/debug/etc...I think it's useful when a lot of options are involved.)

Thanks.

[akupenguin]

x264 `cat optionsfile`

[Sharktooth]

does that work for win32 too?

[omion]

Nope. Unfortunately, it's a *NIX thing (and presumably cygwin)... I wish Microsoft would make their command prompt more useful.

[Sharktooth]

They will in Longhorn server... but i guess it will never be comparable to bash, zsh, etc...

[miztadux]

OK, thanks for your answers to a poor "Posts: 4" guy...
If I may comment a little more on the subject:

x264 `cat optionsfile`
Thanks for the suggestion, but I don't think it's as flexible as the mkvtoolnix option (possibility to add comments, unquoted/unescaped special strings, line feeds instead of spaces, etc...)

Nope. Unfortunately, it's a *NIX thing (and presumably cygwin)... I wish Microsoft would make their command prompt more useful.
Right, there's no alternative in cmd.exe to the "execution quotes" in *sh (and cygwin doesn't help by itselft either as long as cmd.exe is used)

But in this case (@optionfile) i thought the limitation comes from the code used to parse the command line (if you use a library that doesn't support this kind of thing you're good to re-implement the parsing by yourself...)

Anyway I asked in case it could be done easily, the command line options are good enough (especially if you only specify what's different from the default values, you end up with not-that-much options)...

BTW thanks for the efforts on this awesome encoder !!

[virus]

Right, there's no alternative in cmd.exe to the "execution quotes" in *sh
The correct name for that (invaluable) *sh function is "command substitution".



But in this case (@optionfile) i thought the limitation comes from the code used to parse the command line

The current parsing code is OK, as long as you preprocess the options-file a little bit. Unsurprisingly, this is better handled using standard cmdline tools instead of pure C code. Here's a way to input an optionsfile with newlines and comments (by "comment" I mean full rows starting with #):


x264 $(echo `sed 's/^#.*$//' optionsfile`)


This works under a normal *nix shell.
Might not work under MSYS/Cygwin, due to their somewhat strange handling of (back)slashes. You might need to escape the slashes above.

[akupenguin]

x264 $(echo `sed 's/^#.*$//' optionsfile`)
is equivalent to
x264 `sed 's/^#.*$//' optionsfile`
and those are forwardslashes, which are not special.

possibility to add [...] line feeds instead of spaces
Linefeeds are just another kind of whitespace, and can separate words on the commandline the same way that spaces and tabs do. You can't normally type a linefeed on the commandline because it would run your command instead, but it doesn't need any special treatment within a command substitution.

Also, if you have the cygwin/msys utilities but are not using a real shell, then an alternative is xargs x264 < optionsfile

[virus]

indeed, that was a typo. Slashes might create problems, too. I've already had enough problems using sed in MSYS, I'm certainly not trying it further.
The two cmdlines aren't really equivalent, but they indeed both work. So much for forgetting about IFS :)

By the way, this is even shorter:

x264 `sed '/^#/d' optionsfile`

xargs is nice, but won't strip the comments.

[squid_80]

x264 `cat optionsfile`
Stupid win XP equivalent:
set /p x264opts= < optionsfile
x264 %x264opts%
set x264opts=

[omion]

I have an additional request for x264:
When I encode the second pass with a very low bitrate, x264 will occasionally spit out something like:
x264 [error]: requested bitrate is too low. estimated minimum is 3 kbps
Fair enough. It's telling me that the bitrate can't be reached.

However, I have two issues with the current implementation:
I will commonly get the message that says the minimum is the same bitrate that I specified. Like this:
E:\DVD\MKV\ZZZ TESTS# x264 --pass 2 --bitrate 3 --stats blackness.txt -b 7 --progress -o blackness2.mkv blackness.avs
avis [info]: 640x480 @ 24.00 fps (10000 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [error]: requested bitrate is too low. estimated minimum is 3 kbps
x264 [error]: x264_encoder_open failednote that I tried to encode to 3 kbps...

The other, bigger, problem that I have with it is the encoder will simply exit after the message. I think it should be treated the same way as the "2pass curve failed to converge" message; a warning and then a best-effort encoding.


The only time I run into this message is when I try to encode a series of black frames. Although generally encoding blackness isn't too worthwhile, my x264farm program splits up a movie into scenes, some of which may be completely black.


Thanks!