Split a stats file

[MatLz]

Hi guys!
Is possible to split a X264 stats file and get the average bitrate needed for each parts?

[Sagekilla]

If you know a bit of perl (Or any scripting language like Python, etc) you can write a very simple program to do this.

It would be more or less getting the frame count, summing up the bit counts, and doing simple arithmetic. That's really the only way you can. x264 only supports writing to one stats file.

[MatLz]

Thanks Sagekilla.
If it's the only way....can you post a link to these 'scripting language' homepage or tutorials? I'm not aware about.

[Sagekilla]

If you run linux, perl (http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=vJm&q=perl+tutorial&aq=f&oq=&aqi=g10) is probably already available. Otherwise, if you're on windows there's always the option of using IDLE (Python GUI). If you're really clueless, let me know.

Google is your friend (http://www.google.com/search?q=python+idle&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a).

[foxyshadis]

When I used to fansub, I ended up splitting the finished encode and re-encoding segments all the time. I had it down to a science, it was awesome. Unfortunately, I don't have my main work folder with all of the mmg command lines anymore, all I have is the code (in PHP, easily translateable to any other language, but I hate Perl) to split the stats file; if that's enough, I'll post it. If you know programming it isn't hard to modify it for bitrate, but if not, you'll need to specify exactly what you need and how it should output so someone can make it. (Back then, I just copied the bitrate of the segment that I already had.)

[MatLz]

Tanx for your answer Foxyshadis.
Unfortunetly my first pass has crashed at 93% after 33h of encoding. :(
To be clear, when I've ask for splitting stats file, it's because I saw I was really really near to be out of memory cause Multiplesssss happened files and filtering..... So I was afraid to do the second pass in these conditions.
I resolve my problem with happening videos before Avisynth.
Hm...No, I don't know programming...., maybe in the future when the chickens will have teeth! ;)

[MatLz]

To be serious, if it's really a simple programming, maybe I will can do that. Soon I will post a small stats file with qp file and see your method...If you want of course...

[Sagekilla]

If the reason for you wanting to split an encode (or stats file) is because of crashing, then it seems like we should be fixing the crashing rather than trying to just get a workaround here.

Can you provide more details? Avisynth script, computer config, x264 settings used, etc. It would be better to fix the crashing (and possibly, if one exists, any bugs present) rather than just sidestepping the issue.

[MatLz]

@Sagekilla
Thanks, but the crash is not the problem, I was out of memory, it's my fault to have not prepare my sources correctly.

@Foxyshadis
Here are a stats file and the qp file used for indicate cutting. It's just an example for splitting into 3 parts to see your method.
So my need is to have 3 stats files (with, of course, the same allocation of bitrate for each scene than the entire stats file) and the average bitrate for each. Thanks.

[MatLz]

Well, I have the time, I can wait more.
To 'revive' this thread I have a question, maybe out of topic, but linked a little...:
If I can't split the stats file, I can split the video, but datas like codec settings used are lost except for the first part.
Is there a way to add these datas to the others?

[hxhxd]

I don't think it's easy to split a stats file, unless you find a IDR position in the stats file.

[MatLz]

My thread is not really a success...
Ok...so I ask more precise thing, what are, in the stats file:

tex, mv, misc, imb, pmb and smb ?

Are they indicate the quantizer's deviation calculated at the end of the the pass? (Sorry if I'm totally wrong, I'm principaly here to learn...)

[ACoolie]

countquant_x264.pl in x264.git/tools/ or here (http://akuvian.org/src/x264/countquant_x264.pl) documents the stats file format. If all you want is average bitrate, it can figure it out for you.

[MatLz]

Thanks ACoolie.
I downloaded countquant_x264.pl. It seems it's a Perl tool so I downloaded Perl too....but I've really no idea how to use that. :(

And I saw countquant_x264.pl is 4.5 years old...I can see in this file mv and misc but there are itex, ptex...not simple tex...
I'm really lost...

[MatLz]

Well, I just discover imb+pmb+smb=number of macroblocks (except for b frames) so I think we don't care them for my need...
Remain: tex, mv and misc.

Edit: A thing suddenly hit me....it's maybe an other stupid question, but, is the quantizer in the stats file already adapted to give the bitrate we want? Or the deviation is given by the other values (tex, mv, misc) ?

[MatLz]

Not to annoy them, but it seems even the x264 developpers are not able to answer... :(

[Boolsheet]

Careful! If you hit the reply button too hard in quick succession the thread tilts and you lose your ball. http://forum.doom9.org/images/smilies/tongue.gif


I don't know the x264 code at all, so I can just assume what the numbers mean. Here's what I think:

tex is the size of the residual (difference/error whatever you call it)
mv is the size of the motion vectors
misc is the size of... miscellanious?
Add those 3 together and you should get the frame size in bits.

You can edit the perl file to get it working again, just change "itex:(\d+) ptex:(\d+)" to "tex:(\d+)" on line 11 and delete "+$6" on line 17. Then run it with "perl countquant_x264.pl t.stats".
Or you could try lua. I wrote a simple parser (http://pastebin.com/m9effdcc). Get the binaries (http://luabinaries.luaforge.net/) and start it with "lua -i x264statsparser.lua".

[MatLz]

Boolsheet, Thanks for informations and links. I will take an attentive look at that!

[MatLz]

Here is my "C:\lua" directory:

<Microsoft.VC80.CRT>
bin2c5.1.exe
test.stats
lua5.1.dll
lua5.1.exe
lua51.dll
luac5.1.exe
wlua5.1.exe
x264statsparser.lua

I tried lot of combinaisons of command-lines but I'm not able to do it work.
I've often a "unexpected symbol near" error. And one time, which "maybe did work a little", a "unexpected symbol near 'in' " error. Is the "in:" in the stats file?
What exe must I launch and what is the exact commandline in function of my directory?

[Boolsheet]

It's "lua5.1.exe -i x264statsparser.lua".
If lua reports an error after loading, make sure you downloaded the script correctly. (The download button on pastebin worked for me)
The -i switch means interactive mode. It lets you write lua directly into the interpreter. To start the parsing write printStats("test.stats").

If you get any errors while parsing, post the full error message.

[MatLz]

Ok, Thanks. It does launch now, but :



C:\Documents and Settings\Mat>C:\lua\lua5.1.exe -i C:\lua\x264statsparser.lua
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
Usage: getStats(fileName, [startFrame], [endFrame])

> getStats("C:\lua\test.stats", 0, 2100)
stdin:1: attempt to call global 'getStats' (a nil value)
stack traceback:
stdin:1: in main chunk
[C]: ?
>



For 'printstats' I get 'file locked or not found'
:(

[Boolsheet]

Oh, that's what I get for changing function names in the last minute. :)
It's printStats and you have to escape the backslash in strings: "c:\\lua\\test.stats".

[MatLz]

Unfortunetely...same thing...'file locked or not found'...

[Boolsheet]

Uhm, I'm not sure what's wrong. To get a more accurate error message you can change this on line 68:

local statsFile = ioopen(filePath, "rb")
if not statsFile then
return nil, "File locked or not found"
end
to
local statsFile, fileError = ioopen(filePath, "rb")
if not statsFile then
return nil, fileError
end

[MatLz]

Arff...really sorry BoolSheet...it was because I'm on a phone to surf, and on my little screen '\\' appears more like that: '\ \' ...
Sorry again.

Well now it turn. But the result is not the good...it gives me 34.2 kbps instead of the 1206kbps (stream splited)
If I use it on all the frames it gives me 34.95 instead of the 850.

[MatLz]

Sorry again because in fact I used on an 'edited' file (a line for each frame is more readable...).

So the right numbers are:

All frames:
837.94 instead of 850

Part1(0,3042):
818.79 instead of 1206

Part2(3043,6943)
827.8 instead of 723

Part3(6944,11600)
858.96 instead of 723

[MatLz]

First, thx to Boolsheet, even if his tool didn't work, he has been the only guy who really tried to help.

[MatLz]

Is my humble request impossible? (at least without bucks on the table:D)

Let say I encode 3 videos with same crf+same settings, it gives me:
v1=400kbps
v2=900kbps
v3=500kbps
So an average of 600kbps
(to simplify you what I mean, the example videos have same length)

Let say my targeted average bitrate is 1000kbps,
Can I do a 2pass encode with the same proportionnality of bitrate repartition of the crf? So:
v1=666kbps
v2=1500kbps
v3=833kbps

Is crf a good indication to do things like that?