Estimate complexity of video using values from stat file

[Atak_Snajpera]

I'm trying improve bitrate distribution in 2-pass in my Distributed Encoding mode. Currently each chunk (by default 10 min) gets the same bitrate budget. So chunk with dynamic scenes gets the same bitrate budget like chunk with credits.

In order to fix this I'm trying to find a way to estimate which chunks should get more bitrate and vice versa.

So I need someone to help me understand those values stored in stat file.

in:0 out:0 type:I dur:2 cpbdur:2 q:28.04 aq:10.40 tex:4808 mv:1227 misc:6685 imb:1131 pmb:0 smb:0 d:- ref:;
in:3 out:1 type:P dur:2 cpbdur:2 q:28.04 aq:14.03 tex:26 mv:7 misc:311 imb:0 pmb:1 smb:1130 d:- ref:0 ;
in:1 out:2 type:b dur:2 cpbdur:2 q:28.04 aq:13.00 tex:0 mv:0 misc:272 imb:0 pmb:0 smb:1131 d:- ref:0 ;
in:2 out:3 type:b dur:2 cpbdur:2 q:28.04 aq:14.03 tex:26 mv:2 misc:284 imb:0 pmb:1 smb:1130 d:- ref:0 ;
in:4 out:4 type:i dur:2 cpbdur:2 q:25.79 aq:25.43 tex:133370 mv:49792 misc:262 imb:1131 pmb:0 smb:0 d:- ref:;
in:5 out:5 type:P dur:2 cpbdur:2 q:28.04 aq:27.71 tex:112928 mv:9057 misc:351 imb:616 pmb:506 smb:9 d:- ref:0 ;
in:6 out:6 type:P dur:2 cpbdur:2 q:28.04 aq:27.76 tex:110475 mv:10223 misc:326 imb:490 pmb:631 smb:10 d:- ref:0 ;
in:7 out:7 type:P dur:2 cpbdur:2 q:28.04 aq:27.67 tex:91191 mv:8608 misc:393 imb:445 pmb:663 smb:23 d:- ref:0 ;
in:8 out:8 type:P dur:2 cpbdur:2 q:28.04 aq:28.14 tex:108135 mv:9790 misc:371 imb:357 pmb:751 smb:23 d:- ref:0 ;
in:9 out:9 type:P dur:2 cpbdur:2 q:28.04 aq:27.50 tex:101665 mv:8661 misc:346 imb:537 pmb:591 smb:3 d:- ref:0 w:6,63,1 ;

On which values should I focus the most?

[fvisagie]

Although it does not directly answer your question at this stage, if you want the result of distributed rate control to approach that of e.g. x264 on its own, make sure you have a look at http://git.videolan.org/?p=x264.git;a=blob_plain;f=doc/ratecontrol.txt;hb=HEAD.

[kabelbrand]

Another 'indirect' answer: what about running a complete first pass on one worker only and using the 'final ratefactor' value displayed afterwards as crf instead of bitrate for all segment encodes?
Wouldn't this result in the same overall bitrate but distributed more context aware?

[Atak_Snajpera]

Too big encoding speed penalty even with fast first pass. I think I will focus on tex and mv values in order to estimate complexity of chunk.

[sneaker_ger]

Using "final ratefactor" for --crf is not precise enough anyways. 2pass bitrate allocation is more complex than that.

[foxyshadis]

You can just run the first passes in crf mode, then use each chunk's size compared to the combined size to calculate its bitrate. Easy as pie.

For advanced analysis of the stats file you can assume that tex: will change much more than anything else if you're wildly changing the bitrate, so if you've got one chunk where textures are only 50% of the rate and in every other chunk they're 90% or so (typical), then you can assume that when lowering the bitrate the one chunk needs to keep more bits than the others. But the slow second pass will change allocations anyway, you may as well just go with the basic ratio.

[Atak_Snajpera]

You can just run the first passes in crf mode, then use each chunk's size compared to the combined size to calculate its bitrate. Easy as pie.
I've already tried this. There are two problems with this method. Accuracy changes with selected crf value. This graph represents bitrate distribution (bitrate multipliers for each chunk) for 10 min "Drive" movie. Each chunk is 1 min long.
http://i.cubeupload.com/AgWJYG.png

Another more serious problem is that stat file created in CRF mode is not fully compatible in second pass. I've noticed sudden quality drops (not related with bitrate). Those drops do not occur with normal --bitrate command in first pass.

Currently I'm testing new method based on tex: and mv: values. Average value from tex and mv looks promising.
http://i.cubeupload.com/DymY2l.png

[benwaggoner]

We had to do something similar with VC-1 for 1080p BD/HD-DVD with PEP/CineVision PSE. We did it by distributing the chunks to each worker node, then doing a first pass on each. Then the controller collected the stat data and figured out what overall distribution was required, and then told each worker node to encode appropriately on its own chunk. We also did some tweaking around the stitch points to make sure that they were aligned with scene changes etcetera, and wouldn't violate VBV.

Would something like that work?

[Atak_Snajpera]

I guess you are doing the same as I now
http://forum.doom9.org/showthread.php?p=1729404#post1729404

Additionally I have modifiable strength value

http://i.cubeupload.com/KuwPsh.png