Can x264 set environment variables for me?

[UltraNerd]

When encoding using x264 I use a 'crf' first pass to determine my target bitrate. A next pass will use the achieved bitrate of the first pass as '--bitrate' value.

Currently I run the first pass, and manually copy the achieved bitrate before starting the next pass. I would like to write a batchfile that performs all passes without user interaction.

This would be possible if x264 would offer a mechanism to store results of a pass in environment variables, but I have not been able to find (information on) such a feature. I think this would be a nice addition, not only for target bitrate but also for other numbers (ssim/psnr?).

My question is: do you have any tips how to solve this?
Perhaps I am not aware of another solution to achieve the same result? Please advise :)

The smartest trick I can think of is to 'grep' the output of the crf pass to retrieve the achieved bitrate, extend the batchfile to store the grepped result in an environment variable and proceed from there. As a Windows user, grep is not available for me, so I have to look for other tools (which undoubtedly exist, but that is not my point).

[DarkZell666]

Now what exactly are you doing ? What's the use of doing a crf to determine a bitrate, and use that exact bitrate ? Why not just do single-pass crf ? What do you hope to gain by doing this ?

CRF is already smart enough as it is, you'll only get worse quality using ABR mode at the same bitrate.


Edit: the way you collect the CRF value is the only feasible way ATM anyway.

[foxyshadis]

No, running command-line mediainfo over the output would work as well. But any way you do it, something's going to have to be customized to set the given environment variables; might as well use a win32 grep (http://unxutils.sourceforge.net/) for it.

My reason for doing this is to run a fast 2% compressibility pass to make sure it's reasonable (not too high or low), then a full two-pass. I waver between two pass and crf, not entirely sure why.

[Marsu42]

As a Windows user, grep is not available for me, so I have to look for other tools (which undoubtedly exist, but that is not my point).

There are several grep cli implementations for Windows, either w/ cygwin, mingw32 or native win32 - google helps...

[UltraNerd]

Now what exactly are you doing ? What's the use of doing a crf to determine a bitrate, and use that exact bitrate ? Why not just do single-pass crf ? What do you hope to gain by doing this ?

CRF is already smart enough as it is, you'll only get worse quality using ABR mode at the same bitrate.


First pass (crf) generates a stats file, then I use nth pass to read that stats file. This is a technique that was suggested in the online guides I read. And it makes sense to me, I think it is like doing a two-pass encode. An n-th pass is better able put the available bitrate to good use than a first pass is. And PSNR is always through the roof like this, so I like it :)


No, running command-line mediainfo over the output would work as well. But any way you do it, something's going to have to be customized to set the given environment variables; might as well use a win32 grep (http://unxutils.sourceforge.net/) for it.


I did not think about using 'post-processing' to get the numbers I need. This is an interesting option because the solution with grep has a major disadvantage. That solution means you have to pipe the x264.exe output to a file, which means you can't see your progress by looking at the shell output.


Still, I think it would be a great feature to let x264.exe output some statistics in environment variables. Will save people the effort of using additional tools for customizing the output to their needs. I would also use this when running many test-renders to see the effect commandline effects have.

[foxyshadis]

Are you familiar with C at all? Because it wouldn't be a terrifically difficult patch, and sharktooth would probably even accept it if it was proven stable, but it'll be a hard sell to get other developers to devote time to it.

[DarkZell666]

First pass (crf) generates a stats file, then I use nth pass to read that stats file. This is a technique that was suggested in the online guides I read. And it makes sense to me, I think it is like doing a two-pass encode. An n-th pass is better able put the available bitrate to good use than a first pass is. And PSNR is always through the roof like this, so I like it This is in fact very interesting, I didn't know this actually worked, but I'm still trying to make some sense out of it ...

@foxyshadis: I got your point about "finding something reasonable to work with" (it's this DABR topic thing isn't it?). I agree with that. But here, UltraNerd uses the bitrate "as-is", so here goes:

Example: If you use crf 20, and that the bitrate comes out to be 800kbps. If you do the second pass with --bitrate 800, what's the second pass going to do ? reallocate bits differently from what CRF did ? It'll be cleaner than the original 2-pass method, but I'm still to understand how the quality could be "better" than the 1-pass CRF encode. And you forget about the predictable filesize, because you asked for CRF in the first place (unless you use the DABR technique, doing several 2% CRF encodes until the target bitrate matches what you wanted, but this is not what he seems to be doing oO).

UltraNerd, I'm not saying that you don't know what you're doing (far from it), I'm asking myself what I could have missed to understand what you're doing ;)

[shon3i]

@DarkZell666, to me this have some sense, and this look like full quality first pass option in xvid, wich can improve some little quality.

[quake74]

If you are working in WinXP the "for" command provides a very basic grep substitute. Here is how I find out width height and number of frames of an avs file inside a bat file:


x264.exe "%~f1" --frames 1 --output NUL 2>"%~dpn1.stats"
for /f "tokens=3,4,8,9 delims=x() " %%F in (%~dpns1.stats) do if "%%I"=="frames" (
SET WID=%%F
SET HEI=%%G
SET FRAMES=%%H
)

[UltraNerd]

Are you familiar with C at all? Because it wouldn't be a terrifically difficult patch, and sharktooth would probably even accept it if it was proven stable, but it'll be a hard sell to get other developers to devote time to it.

I did not know Sharktooth might accept external patches!! And because I did not want to build my own executables for every new version, I decided not to make changes to the code but search for another solution. If Sharktooth would accept my changes, I'd be happy to look into it! I'll ask as soon as I know how to reach him.



Example: If you use crf 20, and that the bitrate comes out to be 800kbps. If you do the second pass with --bitrate 800, what's the second pass going to do ? reallocate bits differently from what CRF did ? It'll be cleaner than the original 2-pass method, but I'm still to understand how the quality could be "better" than the 1-pass CRF encode. And you forget about the predictable filesize, because you asked for CRF in the first place (unless you use the DABR technique, doing several 2% CRF encodes until the target bitrate matches what you wanted, but this is not what he seems to be doing oO).

UltraNerd, I'm not saying that you don't know what you're doing (far from it), I'm asking myself what I could have missed to understand what you're doing ;)

Please do speak out if you think my approach does not make sense!! I might very well not know what I'm doing here, I've only started looking into h264 encoding a few weeks ago. What you describe is *exactly* what I'm doing.

Obviously I can not answer your question about how bits will be reallocated differently on the pass after the CRF pass. I seriously hope someone with that knowledge will answer that question, as I am very curious myself now. Does the described approach make sense, and why or why not?

What I can tell is that when I compare q-values in the stats-files for a first and second pass I notice a large amount of frames will have a slightly different value (about 55-65%). Those differences are much less between second and third pass (about 15-20%). These numbers have been gathered on large files (>60K frames).

I decided on this approach based on pieces of text on these pages:

http://www.mplayerhq.hu/DOCS/HTML/en/menc-feat-x264.html
claiming among others:
"using two pass often gains around 1dB PSNR, which is a very big difference"
and
"single pass ratecontrol is not psychic, and it often makes unreasonable choices because it cannot see the big picture"

http://aflux.deltaanime.net/Zero1/MP4/x264.html
claiming among others:
"2 pass usually brings about higher quality [...] and is always recommended"





x264.exe "%~f1" --frames 1 --output NUL 2>"%~dpn1.stats"
for /f "tokens=3,4,8,9 delims=x() " %%F in (%~dpns1.stats) do if "%%I"=="frames" (
SET WID=%%F
SET HEI=%%G
SET FRAMES=%%H
)


I am impressed. And here I was thinking I knew a fair bit about batch files :)
Do I understand the code correctly when I say the for loop is only needed in cases where the numbers you are looking for are not on the first line of the output (which they always are as far as I know)?
It would be easy to change this code for my particular problem, I would need to look for a line template like this:
"encoded %a frames, %b fps, %c kb/s"

[galaxy001]

If there is going to be a patch, I hope a percent can be set.
Such as we can use 0.95 of the crf bitrate in 1-pass.
This will be more useful, right?

[quake74]

I am impressed. And here I was thinking I knew a fair bit about batch files :)
Do I understand the code correctly when I say the for loop is only needed in cases where the numbers you are looking for are not on the first line of the output (which they always are as far as I know)?
It would be easy to change this code for my particular problem, I would need to look for a line template like this:
"encoded %a frames, %b fps, %c kb/s"

First of all, the best reference I found for batch files is http://www.allenware.com/icsw/icswidx.htm. In your case, you could just write


for /f "tokens=1,2,4,6 delims= " %%F in (filenamein83format) do if "%%F"=="encoded" (
SET ENC=%%G
SET BPS=%%H
SET KBS=%%I
)

[akupenguin]

If there is going to be a patch, I hope a percent can be set.
Such as we can use 0.95 of the crf bitrate in 1-pass.
Just use a higher value of crf?

[galaxy001]

for /f "tokens=1,2,4,6 delims= " %%F in (filenamein83format) do if "%%F"=="encoded" (
SET ENC=%%G
SET BPS=%%H
SET KBS=%%I
)


This script will get a float number which is not supported by x264 core:54 svn-616 @ http://mirror01.x264.nl/x264/revision616/x264.exe.

Here is mine:

for /f "eol=x tokens=2,3,7 delims=,. " %%F in (.\work\264log.txt) do if "%%G"=="frames" (
SET _frames=%%F
SET _bps=%%H
)

[UltraNerd]

I ended up solving my problem by using tee.exe from the UnxUtils package. This captures all output from the commandline to a file, while still writing it to the shell.

The batchfile that calls x264.exe parses the file-output file to retrieve the bitrate.

:thanks: Thanks to all for your comments and insights!

[Single]

When encoding using x264 I use a 'crf' first pass to determine my target bitrate. A next pass will use the achieved bitrate of the first pass as '--bitrate' value.


I have an idea to do it differ way just before read you post. Since a 2-pass encoding always will have better quality I think to do a two pass encoding after a 'crf'.

When no target bitrate are known 'crf' determine it at quality that is suitable for me. And 2-pass encoding gives an extra quality at the same bitrate.

What dev have to say about this idea?

[DarkZell666]

When no target bitrate are known 'crf' determine it at quality that is suitable for me. And 2-pass encoding gives an extra quality at the same bitrate.I don't agree :D

1-pass CRF gives better quality at same average bitrate than 2-pass IMHO. This thread didn't spawn for nothing: http://forum.doom9.org/showthread.php?t=116773&highlight=DABR

Well, "better" meaning more "constant" than 2-pass. But then you can still tweak the qcomp parameter and get a similar result in 2-pass, except that you actually needed 2 passes instead of 1 :p The whole point of the thread was how to get a freaking-good quality 1-pass encode (and it works ;)).

AutoMKV sort of implements the technique, and someone posted a tool in the above-mentioned thread for testing it.

[check]

That thread seems to suggest 2pass is better: http://forum.doom9.org/showthread.php?t=116773&page=3&highlight=DABR#50

[DarkZell666]

We must have read what's in there differently then :p

For me, doing a 2-pass encode with a high qcomp is pretty equivalent to doing a 1-pass CRF encode (at samey average quant.). The preference goes to the compromise you make between hi-motion and lo-motion scenes :)

When you've got a limited number of bits to allocate to a given movie, the only thing that changes is WHERE the bits are allocated, and the only thing that distinguishes the bitrate allocation between 2-pass@qcomp>0.80 and CRF is the name and your being nervous because you drunk two coffees instead of one ;)

I suppose we're falling back to the "what's best" question again :rolleyes:

[foxyshadis]

There's genuine reasons 2-pass is actually better: Better I-frame handling and better b-direct mode.

But by how much is the question, and it's very, very small when qcomp is high.