x264: First IDR frame of a CBR encode looks horrible

[Guest]

I'm encoding CBR for transport stream broadcast, so please don't tell me don't use CBR. :)

I find that when I encode in CBR mode (see below), the first IDR frame looks horrible. I would like to know why it happens (vbv init issue?) and how to fix it. I used a fairly small bitrate for this material to show the problem really well, but it happens at higher bitrates too and it's a static scene, so bitrate shouldn't be that relevant anyway.

Here is the encode. Open it in DGAVCIndex and you'll see that the first IDR frame is horrible. Subsequent ones are fine. There's no reason the first IDR should be any worse.

http://neuron2.net/misc/cbr5Mprog.zip

Here are the settings used:

-----
x264 core:65 r1071 16c8553

E:\tmp\x264 Encoding Test>x264 --keyint 25 --min-keyint 1 --bframes 3 --b-adapt 2
--ref 3 --bitrate 5000 --vbv-maxrate 5000 --vbv-bufsize 5000 --ratetol 0.1 --qcomp 0.0
--direct "auto" --weightb --mixed-refs --8x8dct --no-fast-pskip --output "cbr5Mprog.264"
--fps 23.976 --progress --no-psnr --no-ssim --threads auto --aud "Baraka.avs"
avis [info]: 1920x1080 @ 23.98 fps (1786 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4.1 Cache64
x264 [info]: profile High, level 4.0
x264 [info]: slice I:73 Avg QP:20.87 size:203270
x264 [info]: slice P:621 Avg QP:21.47 size: 41090
x264 [info]: slice B:1092 Avg QP:24.40 size: 5842
x264 [info]: consecutive B-frames: 8.5% 10.0% 28.2% 53.2%
x264 [info]: mb I I16..4: 13.1% 72.7% 14.1%
x264 [info]: mb P I16..4: 1.7% 1.4% 0.1% P16..4: 47.7% 12.3% 14.9% 0.0% 0
.0% skip:21.9%
x264 [info]: mb B I16..4: 1.9% 0.0% 0.0% B16..8: 14.6% 0.3% 0.2% direct:
8.2% skip:74.8% L0:26.3% L1:56.3% BI:17.4%
x264 [info]: 8x8 transform intra:54.4% inter:51.4%
x264 [info]: direct mvs spatial:99.5% temporal:0.5%
x264 [info]: ref P L0 77.6% 15.3% 7.1%
x264 [info]: ref B L0 81.3% 18.7%
x264 [info]: kb/s:5019.1

encoded 1786 frames, 8.59 fps, 5020.84 kb/s
-----

I can upload the source if it is needed. Any advice will be appreciated.

[Dark Shikari]

--ratetol 0.1This is probably your problem. Don't adjust ratetol from the default value without good reason--doing so can make x264 far too overzealous in trying to hit your target bitrate even in just the first few frames, resulting in what you saw there. --qcomp 0.0Don't touch qcomp without good reason.

Also, if you're not doing a realtime encode, you should use twopass if possible.

[Guest]

I need a good CBR for broadcast. My understanding was that you need --ratetol 0.1 --qcomp 0.0 to avoid bitrate spikes.

In a broadcast environment where you have multiple programs, the individual VBV models are not sufficient to avoid overflow of the transport rate, so a good CBR on each program is needed. Then the transport rate is set to the sum (or slightly greater) of the individual maximums.

If x264 can hit my target bitrate in the second and subsequent IDRs, why can't it hit it for the first? After all, an IDR refreshes all state.

[Dark Shikari]

I need a good CBR for broadcast. My understanding was that you need --ratetol 0.1 --qcomp 0.0 to avoid bitrate spikes.Those have nothing to do with bitrate spikes.

In fact, testing at Avail showed that too low a qcomp could actually increase the odds of violating VBV. Currently, for broadcast, I think Avail uses 0.5 or 0.6 (the default).

Though, nowadays, x264's VBV model should be compliant enough for most purposes without any modification.If x264 can hit my target bitrate in the second and subsequent IDRs, why can't it hit it for the first? After all, an IDR refreshes all state.IDR frames do not reset the buffer state.In a broadcast environment where you have multiple programs, the individual VBV models are not sufficient to avoid overflow of the transport rate, so a good CBR on each program is needed.OK, now I'm confused. "A good CBR" is your VBV model.

[Guest]

IDR frames do not reset the buffer state. Good point. :)

Please comment on my point about individual VBV models not being sufficient to control overall transport rate.

Also, what ratetol does Avail use for broadcast?

[Dark Shikari]

Please comment on my point about individual VBV models not being sufficient to control overall transport rate.I cannot understand what you mean by such a thing. CBR is a VBV model. If you have 5 CBR streams of 5 megabits each, assuming none violate their VBV constraints, you can send them over a 25 megabit link.

This is what Avail does, though Avail allocates a small amount of buffer on the top for the rare case that VBV is violated. This was more important in the past, as before this year's VBV changes x264 would violate VBV up to a few times an hour, requiring a few megabytes of buffer on each satellite transport to handle the possibility.Also, what ratetol does Avail use for broadcast?Default. ratetol has absolutely zero effect on VBV compliance.

[Guest]

Ah, then maybe my muxer is having issues.

I'll try some experiments with your suggestions and report back. Thank you for your advice.

[Dark Shikari]

Ah, then maybe my muxer is having issues.This sounds possible--from my experience at Avail the timing of the muxed packets is extremely nontrivial, especially when one has to deal with the case of a possible underflow by the encoder in addition to the already-complex buffering situation.

[Guest]

OK, I set ratetol and qcomp at default and the first IDR is now fine. I noticed also that I frame pulsing I had seen sometimes is also gone for all practical purposes. :)

There's some disinformation about CBR out there. This thread will be useful to clarify that.

Thanks, Dark. I'll investigate the muxing angle now.

[Dark Shikari]

That's another problem I didn't mention--really low qcomp can lead to I-frame pulsing when combined with VBV, because the buffer will generally stay rather empty, leaving not enough room for I-frames.

At some point I should probably modify the docs to remove the phrase "CBR" as associated with qcomp to avoid misleading people into thinking that they need qcomp 0 to achieve CBR.

[Guest]

Yes, that would be very helpful.

[Sagittaire]

There's some disinformation about CBR out there. This thread will be useful to clarify that.



qcomp at 0 imply in theory no bitrate variance. The encoder try to use 26 Ko for each frame in your case. It's not really good way for Iframe. It's a strict CBR mode.

Anyway in your case you use buffer at 5000 KBits. That's mean that bitrate must be constant only for multiple frame interval and not only for one frame interval. It's not a strict CBR mode.

[Gabriel_Bouvigne]

I'm sorry, but if we are talking about H.264, we should use "CBR" as defined by the H.264 standard. There is no "strict CBR mode". Technically, CBR means that the CPB buffer constraint are enforced, without any overflow or underflow, and nothing else. Any stream complying with its CPB settings (called "VBV" within x264) is CBR, period.

[Sergey A. Sablin]

I'm sorry, but if we are talking about H.264, we should use "CBR" as defined by the H.264 standard. There is no "strict CBR mode". Technically, CBR means that the CPB buffer constraint are enforced, without any overflow or underflow, and nothing else.

Gabriel, CBR means that there is no underflow AND overflow in CPB at any time. To maintain later Filler SEI might need to be used to compensate bitrate variations. Imagine there is a static picture in a stream, which is coded with skip MBs only, and it lasts N seconds - it'll overflow CPB if N > bufsize/bitrate, or even sooner - subject to CPB state atm, and no other trick like per MB quantization will help here.
I haven't heard anything about Filler payload in x264, so I doubt one can claim that CBR is 100% compliant with spec on any sources.

On the other hand H.264 defines VBR as well - in which case only underflow is not allowed. (overflow actually is not possible in VBR case)

Any stream complying with its CPB settings (called "VBV" within x264) is CBR, period.
any explanations?

[Gabriel_Bouvigne]

It seems that we have a non-native language issue ;-)

What I meant is that CBR is CPB enforced, including detection of overflow and underflow ("without any..." => should never occur). My reply was targeted toward the "strict CBR" vs qcomp setting, which could have been interpreted as pointing that there are "strict" and "not strict" CBR settings according to qcomp value, which is not the case. I just don't want that people googling about CBR could think, by reading this kind of threads, that CBR means that every single frame features the exact same number of bits.

Side note: if someone is interested by filler data for x264, a patch was posted to the mailing list a few months ago:
http://mailman.videolan.org/pipermail/x264-devel/2008-April/004389.html

[Sergey A. Sablin]

It seems that we have a non-native language issue ;-)

What I meant is that CBR is CPB enforced, including detection of overflow and underflow ("without any..." => should never occur). My reply was targeted toward the "strict CBR" vs qcomp setting, which could have been interpreted as pointing that there are "strict" and "not strict" CBR settings according to qcomp value, which is not the case. I just don't want that people googling about CBR could think, by reading this kind of threads, that CBR means that every single frame features the exact same number of bits.

Side note: if someone is interested by filler data for x264, a patch was posted to the mailing list a few months ago:
http://mailman.videolan.org/pipermail/x264-devel/2008-April/004389.html

gottcha, sorry for misunderstanding.

Still confused by this statement:
Any stream complying with its CPB settings (called "VBV" within x264) is CBR, period.
does this means that x264 can't produce HRD compliant VBR or I miss something?

[Gabriel_Bouvigne]

"stock" x264 is able to produce HRD compliant VBR streams (although it's not able to embed SEI HRD info)

[Guest]

For CBR, my tests show that stock x264 does not perform bit stuffing to avoid overflow of the VBV model. If x264 just resets the buffer fullness to max on an overflow, then the encoder and decoder buffers are going to get out of sync (leaving aside that it is simply not VBV compliant). I think this is at least one main reason (there may be more) why I get terrible streams when encoding with VBV in x264.

Gabriel, regarding the patch you made last year to implement bit stuffing: Do you have a binary that I could test? Or do you think that patch will still apply to the current codebase?

http://mailman.videolan.org/pipermail/x264-devel/2008-April/004400.html

[akupenguin]

Lack of bit stuffing can't have anything to do with quality problems. If x264 just resets the buffer fullness to max on an overflow, then the decoder has to not read at max bitrate for that interval, and they remain in sync.

[Manao]

I think this is at least one main reason (there may be more) why I get terrible streams when encoding with VBV in x264.What do you mean by terrible ? If you get a bad quality, it's not because of the lack of padding. If a VBR stream isn't correctly played back by some device, I'd say it's the device's fault.

According to the VBV/CPB model, a CBR compliant stream from which you remove any stuffing bytes becomes VBR compliant. So stuffing isn't actually needed.

[Guest]

I'm trying to figure out why my VBV encodes are coming out so crappy. By that I mean I get terrible I-frame flicker that I can't get rid of. And I get VBV underflows reported by x264 that I can't get rid of. The latter should be considered a bug in x264 VBV functionality, IMHO. After I do a few more experiments I'll post a source stream and my settings, so you guys can have a go at it.

First, I thought that as I am getting VBV underflows, that the VBV model was messed up and x264 was thinking it had to overly quantize the I frames because of VBV constraints. So I made a VBV verifier to see what x264 was making. The first thing I saw was VBV overflows. I didn't want to cap at buffer full because that is not strictly VBV compliant for "CBR" encodes.

So I got Gabriel's patch and just built it and have stuffing working so that the VBV overflows no longer happen. I am now experimenting with that to see if it makes any difference.

As I said, I'll post the sample soon. Keep in mind that I need good VBV compliance for broadcasted constant bitrate transport streams with multiple programs.

[Dark Shikari]

I'm trying to figure out why my VBV encodes are coming out so crappy. By that I mean I get terrible I-frame flicker that I can't get rid of. And I get VBV underflows reported by x264 that I can't get rid of. The latter should be considered a bug in x264 VBV functionality, IMHO. After I do a few more experiments I'll post a source stream and my settings, so you guys can have a go at it.Post your settings.First, I thought that as I am getting VBV underflows, that the VBV model was messed up and x264 was thinking it had to overly quantize the I frames because of VBV constraints. So I made a VBV verifier to see what x264 was making. The first thing I saw was VBV overflows. I didn't want to cap at buffer full because that is not strictly VBV compliant for "CBR" encodes.

As I said, I'll post the sample soon. Keep in mind that I need good VBV compliance for broadcasted constant bitrate transport streams with multiple programs.Avail Media does MPTS streams just fine without stuffing in x264, AFAIK.

I see no reason why you need null packet stuff to avoid underflows; the two should be completely unrelated. Perhaps you meant overflows? But x264 doesn't report overflows.

Also, if you have a reliable case of x264 reporting VBV underflows, it's a bug (unless its a case where it is physically impossible to avoid underflow even at QP51, which is unlikely)--report it with all the information necessary to replicate it.

[Manao]

By that I mean I get terrible I-frame flicker that I can't get rid of. Check the quantizers of the I frames that flicker and of their surrounding P frames. If the quantizer of the I-frame is lower than those of the P frames (by 2 or 3, usually), then the VBV didn't hamper x264 and the flickering is due to the I frame itself (and you won't get rid of it without adding some features to x264)

If the quantizer of the I-frame is close or above those of the P frames, then x264 badly handled the VBV. That can be fixed algorithmically. Also, do you test if possible without threads (since threads complicate the rate control job).

By the way, what bitrate / vbv size are you targetting, for what kind of content (complexity, resolution, framerate) ?

Finally, iirc, dark shikari has a patch fixing some problems with VBV, though I don't remember what it actually fixed.

[Guest]

I think this following will tell you everything! The material is basically black until around frame 830, when it cuts to a highly detailed picture and remains as a static image until the end of the trace below (so there should be no flicker). The content is 1080i @29.97 (hard pulldown and encoded interlaced) and very complex (Baraka). Here is the command line:

x264 --keyint 25 --min-keyint 1 --bframes 5 \
--b-adapt 2 --ref 3 --bitrate 10000 --vbv-maxrate 10000 \
--vbv-bufsize 20000 --direct "spatial" --weightb \
--mixed-refs --8x8dct --no-fast-pskip --output "out.264" \
--interlaced --fps 29.97 --progress --no-psnr --no-ssim \
--threads 1 --verbose --aud --filler "Baraka.avs" Here is the trace. After that I give my observations about it.

x264 [debug]: frame= 800 QP=10.00 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=333 bytes
x264 [debug]: frame= 801 QP=10.00 NAL=2 Slice:P Poc:2 I:0 P:2 SKIP:8158 size=50 bytes
x264 [debug]: frame= 802 QP=10.00 NAL=2 Slice:P Poc:4 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 803 QP=10.00 NAL=2 Slice:P Poc:6 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 804 QP=10.00 NAL=2 Slice:P Poc:8 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 805 QP=10.00 NAL=2 Slice:P Poc:10 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 806 QP=10.00 NAL=2 Slice:P Poc:12 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 807 QP=10.00 NAL=2 Slice:P Poc:14 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 808 QP=10.00 NAL=2 Slice:P Poc:16 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 809 QP=10.00 NAL=2 Slice:P Poc:18 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 810 QP=10.00 NAL=2 Slice:P Poc:20 I:1 P:1 SKIP:8158 size=52 bytes
x264 [debug]: frame= 811 QP=10.00 NAL=2 Slice:P Poc:24 I:0 P:2 SKIP:8158 size=55 bytes
x264 [debug]: frame= 812 QP=12.00 NAL=0 Slice:B Poc:22 I:0 P:2 SKIP:8158 size=47 bytes
x264 [debug]: frame= 813 QP=10.00 NAL=2 Slice:P Poc:26 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 814 QP=10.00 NAL=2 Slice:P Poc:28 I:0 P:1 SKIP:8159 size=48 bytes
x264 [debug]: frame= 815 QP=10.00 NAL=2 Slice:P Poc:40 I:0 P:2 SKIP:8158 size=51 bytes
x264 [debug]: frame= 816 QP=12.00 NAL=0 Slice:B Poc:30 I:0 P:1 SKIP:8159 size=45 bytes
x264 [debug]: frame= 817 QP=12.00 NAL=0 Slice:B Poc:32 I:0 P:1 SKIP:8159 size=45 bytes
x264 [debug]: frame= 818 QP=12.00 NAL=0 Slice:B Poc:34 I:0 P:2 SKIP:8158 size=47 bytes
x264 [debug]: frame= 819 QP=12.00 NAL=0 Slice:B Poc:36 I:0 P:1 SKIP:8159 size=45 bytes
x264 [debug]: frame= 820 QP=12.00 NAL=0 Slice:B Poc:38 I:0 P:2 SKIP:8158 size=47 bytes
x264 [debug]: frame= 821 QP=10.00 NAL=2 Slice:P Poc:44 I:0 P:2 SKIP:8158 size=51 bytes
x264 [debug]: frame= 822 QP=12.00 NAL=0 Slice:B Poc:42 I:0 P:1 SKIP:8159 size=45 bytes
x264 [debug]: frame= 823 QP=10.00 NAL=2 Slice:P Poc:46 I:0 P:2 SKIP:8158 size=51 bytes
x264 [debug]: frame= 824 QP=10.00 NAL=2 Slice:P Poc:48 I:0 P:2 SKIP:8158 size=51 bytes
x264 [debug]: frame= 825 QP=10.00 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=332 bytes
x264 [debug]: frame= 826 QP=10.00 NAL=2 Slice:P Poc:8 I:0 P:3 SKIP:8157 size=53 bytes
x264 [debug]: frame= 827 QP=12.00 NAL=0 Slice:B Poc:2 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 828 QP=12.00 NAL=0 Slice:B Poc:4 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 829 QP=12.00 NAL=0 Slice:B Poc:6 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 830 QP=11.02 NAL=2 Slice:P Poc:20 I:6715 P:6 SKIP:1439 size=719614 bytes
x264 [debug]: frame= 831 QP=12.00 NAL=0 Slice:B Poc:10 I:0 P:241 SKIP:7919 size=78 bytes
x264 [debug]: frame= 832 QP=11.11 NAL=0 Slice:B Poc:12 I:0 P:243 SKIP:7917 size=91 bytes
x264 [debug]: frame= 833 QP=11.11 NAL=0 Slice:B Poc:14 I:0 P:242 SKIP:7918 size=84 bytes
x264 [debug]: frame= 834 QP=11.11 NAL=0 Slice:B Poc:16 I:0 P:242 SKIP:7918 size=84 bytes
x264 [debug]: frame= 835 QP=12.06 NAL=0 Slice:B Poc:18 I:773 P:4185 SKIP:1440 size=400651 bytes
x264 [debug]: frame= 836 QP=14.92 NAL=2 Slice:P Poc:24 I:527 P:6215 SKIP:1418 size=195982 bytes
x264 [debug]: frame= 837 QP=17.27 NAL=0 Slice:B Poc:22 I:0 P:3038 SKIP:4851 size=37163 bytes
x264 [debug]: frame= 838 QP=13.79 NAL=2 Slice:P Poc:32 I:725 P:6055 SKIP:1380 size=343923 bytes
x264 [debug]: frame= 839 QP=17.25 NAL=0 Slice:B Poc:26 I:0 P:3522 SKIP:4440 size=30615 bytes
x264 [debug]: frame= 840 QP=17.46 NAL=0 Slice:B Poc:28 I:234 P:5342 SKIP:1895 size=91883 bytes
x264 [debug]: frame= 841 QP=17.48 NAL=0 Slice:B Poc:30 I:1 P:3255 SKIP:4668 size=28234 bytes
x264 [debug]: frame= 842 QP=11.56 NAL=2 Slice:P Poc:42 I:714 P:6069 SKIP:1377 size=490000 bytes
x264 [debug]: frame= 843 QP=16.27 NAL=0 Slice:B Poc:34 I:175 P:5346 SKIP:1574 size=157283 bytes
x264 [debug]: frame= 844 QP=15.89 NAL=0 Slice:B Poc:36 I:321 P:5141 SKIP:1562 size=144438 bytes
x264 [debug]: frame= 845 QP=15.90 NAL=0 Slice:B Poc:38 I:248 P:5170 SKIP:1513 size=163134 bytes
x264 [debug]: frame= 846 QP=15.62 NAL=0 Slice:B Poc:40 I:1 P:3155 SKIP:4451 size=54497 bytes
x264 [debug]: frame= 847 QP=16.52 NAL=2 Slice:P Poc:44 I:116 P:6530 SKIP:1514 size=122179 bytes
x264 [debug]: frame= 848 QP=15.00 NAL=2 Slice:P Poc:46 I:43 P:6163 SKIP:1954 size=117927 bytes
x264 [debug]: frame= 849 QP=21.29 NAL=2 Slice:P Poc:48 I:72 P:3944 SKIP:4144 size=19705 bytes
x264 [debug]: frame= 850 QP=33.52 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=39467 bytes
x264 [debug]: frame= 851 QP=30.25 NAL=2 Slice:P Poc:2 I:918 P:4325 SKIP:2917 size=32406 bytes
x264 [debug]: frame= 852 QP=25.22 NAL=2 Slice:P Poc:4 I:2841 P:3896 SKIP:1423 size=116107 bytes
x264 [debug]: frame= 853 QP=30.87 NAL=2 Slice:P Poc:12 I:255 P:3533 SKIP:4372 size=5193 bytes
x264 [debug]: frame= 854 QP=29.00 NAL=0 Slice:B Poc:6 I:0 P:2 SKIP:8158 size=45 bytes
x264 [debug]: frame= 855 QP=31.27 NAL=0 Slice:B Poc:8 I:0 P:12 SKIP:8148 size=92 bytes
x264 [debug]: frame= 856 QP=30.76 NAL=0 Slice:B Poc:10 I:0 P:9 SKIP:8151 size=70 bytes
x264 [debug]: frame= 857 QP=27.97 NAL=2 Slice:P Poc:14 I:72 P:4029 SKIP:4059 size=12164 bytes
x264 [debug]: frame= 858 QP=25.48 NAL=2 Slice:P Poc:22 I:171 P:5486 SKIP:2503 size=37538 bytes
x264 [debug]: frame= 859 QP=26.13 NAL=0 Slice:B Poc:16 I:0 P:32 SKIP:8128 size=99 bytes
x264 [debug]: frame= 860 QP=30.13 NAL=0 Slice:B Poc:18 I:0 P:41 SKIP:8119 size=145 bytes
x264 [debug]: frame= 861 QP=31.86 NAL=0 Slice:B Poc:20 I:0 P:18 SKIP:8142 size=89 bytes
x264 [debug]: frame= 862 QP=25.05 NAL=2 Slice:P Poc:24 I:197 P:4156 SKIP:3807 size=25918 bytes
x264 [debug]: frame= 863 QP=24.14 NAL=2 Slice:P Poc:32 I:381 P:5401 SKIP:2378 size=53452 bytes
x264 [debug]: frame= 864 QP=26.83 NAL=0 Slice:B Poc:26 I:2 P:103 SKIP:8055 size=237 bytes
x264 [debug]: frame= 865 QP=26.10 NAL=0 Slice:B Poc:28 I:20 P:194 SKIP:7946 size=631 bytes
x264 [debug]: frame= 866 QP=27.26 NAL=0 Slice:B Poc:30 I:0 P:84 SKIP:8076 size=210 bytes
x264 [debug]: frame= 867 QP=21.85 NAL=2 Slice:P Poc:34 I:233 P:6041 SKIP:1886 size=64026 bytes
x264 [debug]: frame= 868 QP=19.82 NAL=2 Slice:P Poc:36 I:32 P:6449 SKIP:1679 size=83736 bytes
x264 [debug]: frame= 869 QP=17.80 NAL=2 Slice:P Poc:42 I:643 P:6140 SKIP:1377 size=179240 bytes
x264 [debug]: frame= 870 QP=21.46 NAL=0 Slice:B Poc:38 I:203 P:1755 SKIP:6107 size=12832 bytes
x264 [debug]: frame= 871 QP=21.43 NAL=0 Slice:B Poc:40 I:2 P:1299 SKIP:6841 size=4782 bytes
x264 [debug]: frame= 872 QP=15.63 NAL=2 Slice:P Poc:44 I:327 P:6515 SKIP:1318 size=237404 bytes
x264 [debug]: frame= 873 QP=18.36 NAL=2 Slice:P Poc:46 I:59 P:3915 SKIP:4186 size=49550 bytes
x264 [debug]: frame= 874 QP=18.47 NAL=2 Slice:P Poc:48 I:215 P:6418 SKIP:1527 size=104667 bytes
x264 [debug]: frame= 875 QP=28.03 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=87342 bytes
x264 [debug]: frame= 876 QP=25.44 NAL=2 Slice:P Poc:2 I:930 P:5496 SKIP:1734 size=62999 bytes
x264 [debug]: frame= 877 QP=24.83 NAL=2 Slice:P Poc:4 I:483 P:5858 SKIP:1819 size=54426 bytes
x264 [debug]: frame= 878 QP=27.40 NAL=2 Slice:P Poc:12 I:217 P:3330 SKIP:4613 size=8605 bytes
x264 [debug]: frame= 879 QP=27.41 NAL=0 Slice:B Poc:6 I:0 P:14 SKIP:8145 size=79 bytes
x264 [debug]: frame= 880 QP=30.84 NAL=0 Slice:B Poc:8 I:1 P:164 SKIP:7994 size=583 bytes
x264 [debug]: frame= 881 QP=31.65 NAL=0 Slice:B Poc:10 I:2 P:41 SKIP:8117 size=170 bytes
x264 [debug]: frame= 882 QP=25.34 NAL=2 Slice:P Poc:14 I:175 P:5201 SKIP:2784 size=29841 bytes
x264 [debug]: frame= 883 QP=23.84 NAL=2 Slice:P Poc:22 I:261 P:5895 SKIP:2004 size=47928 bytes
x264 [debug]: frame= 884 QP=29.50 NAL=0 Slice:B Poc:16 I:1 P:610 SKIP:7541 size=1648 bytes
x264 [debug]: frame= 885 QP=28.16 NAL=0 Slice:B Poc:18 I:4 P:550 SKIP:7600 size=1710 bytes
x264 [debug]: frame= 886 QP=26.78 NAL=0 Slice:B Poc:20 I:1 P:52 SKIP:8107 size=156 bytes
x264 [debug]: frame= 887 QP=23.04 NAL=2 Slice:P Poc:24 I:235 P:5561 SKIP:2364 size=46438 bytes
x264 [debug]: frame= 888 QP=21.07 NAL=2 Slice:P Poc:32 I:456 P:6137 SKIP:1567 size=93668 bytes
x264 [debug]: frame= 889 QP=26.87 NAL=0 Slice:B Poc:26 I:4 P:923 SKIP:7229 size=2040 bytes
x264 [debug]: frame= 890 QP=25.27 NAL=0 Slice:B Poc:28 I:107 P:1488 SKIP:6544 size=6263 bytes
x264 [debug]: frame= 891 QP=23.50 NAL=0 Slice:B Poc:30 I:1 P:267 SKIP:7892 size=663 bytes
x264 [debug]: frame= 892 QP=20.07 NAL=2 Slice:P Poc:34 I:184 P:6331 SKIP:1645 size=81249 bytes
x264 [debug]: frame= 893 QP=17.81 NAL=2 Slice:P Poc:42 I:547 P:6251 SKIP:1362 size=158549 bytes
x264 [debug]: frame= 894 QP=21.60 NAL=0 Slice:B Poc:36 I:4 P:1223 SKIP:6896 size=4280 bytes
x264 [debug]: frame= 895 QP=22.03 NAL=0 Slice:B Poc:38 I:256 P:2742 SKIP:5079 size=13641 bytes
x264 [debug]: frame= 896 QP=21.37 NAL=0 Slice:B Poc:40 I:0 P:938 SKIP:7200 size=3620 bytes
x264 [debug]: frame= 897 QP=15.89 NAL=2 Slice:P Poc:44 I:299 P:6592 SKIP:1269 size=190955 bytes
x264 [debug]: frame= 898 QP=15.46 NAL=2 Slice:P Poc:46 I:93 P:5979 SKIP:2088 size=100873 bytes
x264 [debug]: frame= 899 QP=19.31 NAL=2 Slice:P Poc:48 I:125 P:5033 SKIP:3002 size=41816 bytes
x264 [debug]: frame= 900 QP=27.60 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=95207 bytes
x264 [debug]: frame= 901 QP=25.14 NAL=2 Slice:P Poc:4 I:1013 P:5508 SKIP:1639 size=65463 bytes
x264 [debug]: frame= 902 QP=30.32 NAL=0 Slice:B Poc:2 I:1 P:28 SKIP:8131 size=88 bytes
x264 [debug]: frame= 903 QP=23.37 NAL=2 Slice:P Poc:12 I:676 P:5849 SKIP:1635 size=69672 bytes
x264 [debug]: frame= 904 QP=26.76 NAL=0 Slice:B Poc:6 I:1 P:94 SKIP:8065 size=198 bytes
x264 [debug]: frame= 905 QP=26.10 NAL=0 Slice:B Poc:8 I:8 P:235 SKIP:7917 size=626 bytes
x264 [debug]: frame= 906 QP=25.70 NAL=0 Slice:B Poc:10 I:2 P:84 SKIP:8074 size=249 bytes
x264 [debug]: frame= 907 QP=22.76 NAL=2 Slice:P Poc:14 I:388 P:5844 SKIP:1928 size=61903 bytes
x264 [debug]: frame= 908 QP=21.20 NAL=2 Slice:P Poc:22 I:472 P:6183 SKIP:1505 size=107946 bytes
x264 [debug]: frame= 909 QP=22.99 NAL=0 Slice:B Poc:16 I:1 P:484 SKIP:7673 size=1061 bytes
x264 [debug]: frame= 910 QP=23.73 NAL=0 Slice:B Poc:18 I:136 P:1235 SKIP:6783 size=4886 bytes
x264 [debug]: frame= 911 QP=23.69 NAL=0 Slice:B Poc:20 I:3 P:424 SKIP:7728 size=1186 bytes
x264 [debug]: frame= 912 QP=20.05 NAL=2 Slice:P Poc:24 I:184 P:6364 SKIP:1612 size=84476 bytes
x264 [debug]: frame= 913 QP=19.22 NAL=2 Slice:P Poc:32 I:370 P:6308 SKIP:1482 size=108541 bytes
x264 [debug]: frame= 914 QP=21.66 NAL=0 Slice:B Poc:26 I:7 P:1197 SKIP:6928 size=4248 bytes
x264 [debug]: frame= 915 QP=21.94 NAL=0 Slice:B Poc:28 I:224 P:2750 SKIP:5074 size=17737 bytes
x264 [debug]: frame= 916 QP=21.04 NAL=0 Slice:B Poc:30 I:2 P:1036 SKIP:7104 size=4081 bytes
x264 [debug]: frame= 917 QP=18.05 NAL=2 Slice:P Poc:34 I:158 P:6568 SKIP:1434 size=107672 bytes
x264 [debug]: frame= 918 QP=21.34 NAL=2 Slice:P Poc:42 I:94 P:6222 SKIP:1844 size=64653 bytes
x264 [debug]: frame= 919 QP=22.29 NAL=0 Slice:B Poc:36 I:0 P:636 SKIP:7513 size=1449 bytes
x264 [debug]: frame= 920 QP=22.98 NAL=0 Slice:B Poc:38 I:310 P:1524 SKIP:6252 size=9001 bytes
x264 [debug]: frame= 921 QP=23.35 NAL=0 Slice:B Poc:40 I:38 P:1184 SKIP:6910 size=4088 bytes
x264 [debug]: frame= 922 QP=20.14 NAL=2 Slice:P Poc:44 I:87 P:6071 SKIP:2002 size=62273 bytes
x264 [debug]: frame= 923 QP=18.97 NAL=2 Slice:P Poc:46 I:45 P:5590 SKIP:2525 size=56114 bytes
x264 [debug]: frame= 924 QP=17.79 NAL=2 Slice:P Poc:48 I:395 P:6263 SKIP:1502 size=117375 bytes
x264 [debug]: frame= 925 QP=27.92 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=94242 bytes
x264 [debug]: frame= 926 QP=25.10 NAL=2 Slice:P Poc:2 I:1081 P:5162 SKIP:1917 size=65421 bytes
x264 [debug]: frame= 927 QP=26.36 NAL=2 Slice:P Poc:4 I:360 P:4555 SKIP:3245 size=20973 bytes
x264 [debug]: frame= 928 QP=24.53 NAL=2 Slice:P Poc:10 I:436 P:5722 SKIP:2002 size=48326 bytes
x264 [debug]: frame= 929 QP=24.84 NAL=0 Slice:B Poc:6 I:0 P:59 SKIP:8101 size=137 bytes
x264 [debug]: frame= 930 QP=25.27 NAL=0 Slice:B Poc:8 I:2 P:50 SKIP:8108 size=150 bytes
x264 [debug]: frame= 931 QP=23.08 NAL=2 Slice:P Poc:16 I:548 P:5798 SKIP:1814 size=62033 bytes
x264 [debug]: frame= 932 QP=26.57 NAL=0 Slice:B Poc:12 I:6 P:795 SKIP:7350 size=2075 bytes
x264 [debug]: frame= 933 QP=26.39 NAL=0 Slice:B Poc:14 I:7 P:535 SKIP:7615 size=1745 bytes
x264 [debug]: frame= 934 QP=21.93 NAL=2 Slice:P Poc:22 I:534 P:5738 SKIP:1888 size=67136 bytes
x264 [debug]: frame= 935 QP=25.23 NAL=0 Slice:B Poc:18 I:39 P:1171 SKIP:6924 size=4556 bytes
x264 [debug]: frame= 936 QP=24.64 NAL=0 Slice:B Poc:20 I:0 P:329 SKIP:7831 size=879 bytes
x264 [debug]: frame= 937 QP=20.76 NAL=2 Slice:P Poc:24 I:180 P:6290 SKIP:1690 size=74864 bytes
x264 [debug]: frame= 938 QP=18.96 NAL=2 Slice:P Poc:32 I:399 P:6306 SKIP:1455 size=141756 bytes
x264 [debug]: frame= 939 QP=22.44 NAL=0 Slice:B Poc:26 I:3 P:1292 SKIP:6847 size=4218 bytes
x264 [debug]: frame= 940 QP=22.41 NAL=0 Slice:B Poc:28 I:205 P:1563 SKIP:6328 size=10196 bytes
x264 [debug]: frame= 941 QP=22.68 NAL=0 Slice:B Poc:30 I:2 P:775 SKIP:7370 size=2664 bytes
x264 [debug]: frame= 942 QP=18.05 NAL=2 Slice:P Poc:34 I:193 P:6543 SKIP:1424 size=132152 bytes
x264 [debug]: frame= 943 QP=22.47 NAL=2 Slice:P Poc:44 I:64 P:5402 SKIP:2694 size=32288 bytes
x264 [debug]: frame= 944 QP=24.52 NAL=0 Slice:B Poc:36 I:1 P:1119 SKIP:6995 size=4315 bytes
x264 [debug]: frame= 945 QP=24.05 NAL=0 Slice:B Poc:38 I:237 P:1306 SKIP:6582 size=6386 bytes
x264 [debug]: frame= 946 QP=22.96 NAL=0 Slice:B Poc:40 I:271 P:494 SKIP:7385 size=4336 bytes
x264 [debug]: frame= 947 QP=23.55 NAL=0 Slice:B Poc:42 I:185 P:1265 SKIP:6685 size=5169 bytes
x264 [debug]: frame= 948 QP=20.31 NAL=2 Slice:P Poc:46 I:47 P:5818 SKIP:2295 size=52144 bytes
x264 [debug]: frame= 949 QP=19.77 NAL=2 Slice:P Poc:48 I:171 P:5830 SKIP:2159 size=67752 bytes
x264 [debug]: frame= 950 QP=23.67 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=166339 bytes 1. At frame 835, the first detailed picture (display order) is rendered as a B frame! And it is massive, at over 3 Mbits, because it is still using the quantizer of 12 that suited the black frames. The next P (display order) is over 5Mbits. This bad startup just creams the VBV functionality. I wanted to have a 6MBit buffer, but this just makes that impossible, too.

2. Now look at the subsequent I frames. They have very large quants compared to the preceding P frames. That is the flicker.

3. The first detailed picture should have been an I frame.

[Dark Shikari]

1. At frame 830, the first detailed picture is rendered as a P frame! And it is massive, at over 5 Mbits, because it is still using the quantizer of 11 that suited the black frames. This bad startup just creams the VBV functionality. I wanted to have a 6MBit buffer, but this just makes that impossible, too.

2. Now look at the subsequent I frames. They have very large quants compared to the preceding P frames. That is the flicker.

What can be done?This sounds more like a problem that is not solvable without lookahead--the encoder has no way of knowing that the next scene will be much more complex than the current one--it can only judge from what it knows about the complexity of the current frame.

This does remind me of code I added a while back to limit the size of I-frames; I think the limitation was to 50% of the size of the total remaining VBV buffer.

More heuristics could be introduced to solve this problem in the absence of lookahead, but I've found that most heuristics tend to always have some case in which they make things worse, which always is a problem when trying to minimize problems like this.

I'm going to guess the clip comes out fine with 2-pass?

[Guest]

This does remind me of code I added a while back to limit the size of I-frames; I think the limitation was to 50% of the size of the total remaining VBV buffer. Wouldn't help, as the big picture is not being rendered as an I picture, as it should be.

I'll try two pass and report back.

[Guest]

Two passes cures the flicker, but still, the scenecut should be an I picture. Why isn't it? Off to read the other thread about this...

And my VBV verifier is still going into underflow. I'll look into that and report back.

x264 [debug]: frame= 826 QP=10.00 NAL=2 Slice:P Poc:8 I:0 P:3 SKIP:8157 size=53 bytes
x264 [debug]: frame= 827 QP=12.00 NAL=0 Slice:B Poc:2 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 828 QP=12.00 NAL=0 Slice:B Poc:4 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 829 QP=12.00 NAL=0 Slice:B Poc:6 I:0 P:1 SKIP:8159 size=44 bytes
x264 [debug]: frame= 830 QP=20.29 NAL=2 Slice:P Poc:20 I:6712 P:9 SKIP:1439 size=297051 bytes
x264 [debug]: frame= 831 QP=14.00 NAL=0 Slice:B Poc:10 I:0 P:241 SKIP:7919 size=76 bytes
x264 [debug]: frame= 832 QP=16.00 NAL=0 Slice:B Poc:12 I:0 P:242 SKIP:7918 size=81 bytes
x264 [debug]: frame= 833 QP=18.00 NAL=0 Slice:B Poc:14 I:0 P:243 SKIP:7917 size=77 bytes
x264 [debug]: frame= 834 QP=19.00 NAL=0 Slice:B Poc:16 I:0 P:265 SKIP:7895 size=105 bytes
x264 [debug]: frame= 835 QP=22.26 NAL=0 Slice:B Poc:18 I:748 P:4965 SKIP:2241 size=52675 bytes
x264 [debug]: frame= 836 QP=20.54 NAL=2 Slice:P Poc:24 I:292 P:6368 SKIP:1500 size=87813 bytes
x264 [debug]: frame= 837 QP=22.35 NAL=0 Slice:B Poc:22 I:9 P:2234 SKIP:5881 size=8181 bytes
x264 [debug]: frame= 838 QP=19.56 NAL=2 Slice:P Poc:32 I:249 P:6441 SKIP:1470 size=128775 bytes
x264 [debug]: frame= 839 QP=21.81 NAL=0 Slice:B Poc:26 I:26 P:1383 SKIP:6733 size=5213 bytes
x264 [debug]: frame= 840 QP=22.51 NAL=0 Slice:B Poc:28 I:329 P:3214 SKIP:4505 size=19810 bytes
x264 [debug]: frame= 841 QP=21.52 NAL=0 Slice:B Poc:30 I:8 P:985 SKIP:7141 size=3803 bytes
x264 [debug]: frame= 842 QP=18.52 NAL=2 Slice:P Poc:42 I:378 P:6314 SKIP:1468 size=155014 bytes
x264 [debug]: frame= 843 QP=21.82 NAL=0 Slice:B Poc:34 I:242 P:4008 SKIP:3682 size=33397 bytes
x264 [debug]: frame= 844 QP=21.63 NAL=0 Slice:B Poc:36 I:413 P:3700 SKIP:3813 size=31993 bytes
x264 [debug]: frame= 845 QP=21.64 NAL=0 Slice:B Poc:38 I:320 P:4362 SKIP:3210 size=37955 bytes
x264 [debug]: frame= 846 QP=21.20 NAL=0 Slice:B Poc:40 I:6 P:1934 SKIP:6128 size=8506 bytes
x264 [debug]: frame= 847 QP=18.45 NAL=2 Slice:P Poc:44 I:128 P:6502 SKIP:1530 size=106484 bytes
x264 [debug]: frame= 848 QP=18.41 NAL=2 Slice:P Poc:46 I:34 P:4409 SKIP:3717 size=50296 bytes
x264 [debug]: frame= 849 QP=18.48 NAL=2 Slice:P Poc:48 I:239 P:6336 SKIP:1585 size=100446 bytes
x264 [debug]: frame= 850 QP=17.36 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=396015 bytes
x264 [debug]: frame= 851 QP=19.42 NAL=2 Slice:P Poc:2 I:98 P:3445 SKIP:4617 size=38674 bytes
x264 [debug]: frame= 852 QP=18.50 NAL=2 Slice:P Poc:4 I:87 P:6453 SKIP:1620 size=89550 bytes
x264 [debug]: frame= 853 QP=18.44 NAL=2 Slice:P Poc:12 I:364 P:6267 SKIP:1529 size=126793 bytes
x264 [debug]: frame= 854 QP=19.90 NAL=0 Slice:B Poc:6 I:5 P:2209 SKIP:5885 size=9970 bytes
x264 [debug]: frame= 855 QP=20.24 NAL=0 Slice:B Poc:8 I:296 P:4266 SKIP:3354 size=34308 bytes
x264 [debug]: frame= 856 QP=20.24 NAL=0 Slice:B Poc:10 I:5 P:2504 SKIP:5567 size=12147 bytes
x264 [debug]: frame= 857 QP=18.49 NAL=2 Slice:P Poc:14 I:107 P:6260 SKIP:1793 size=78201 bytes
x264 [debug]: frame= 858 QP=18.44 NAL=2 Slice:P Poc:22 I:362 P:6226 SKIP:1572 size=102788 bytes
x264 [debug]: frame= 859 QP=20.14 NAL=0 Slice:B Poc:16 I:4 P:2616 SKIP:5464 size=10966 bytes
x264 [debug]: frame= 860 QP=20.07 NAL=0 Slice:B Poc:18 I:282 P:3923 SKIP:3805 size=27459 bytes
x264 [debug]: frame= 861 QP=20.22 NAL=0 Slice:B Poc:20 I:4 P:2577 SKIP:5483 size=12536 bytes
x264 [debug]: frame= 862 QP=18.44 NAL=2 Slice:P Poc:24 I:97 P:6016 SKIP:2047 size=68208 bytes
x264 [debug]: frame= 863 QP=19.22 NAL=2 Slice:P Poc:32 I:200 P:5902 SKIP:2058 size=67524 bytes
x264 [debug]: frame= 864 QP=20.15 NAL=0 Slice:B Poc:26 I:3 P:1158 SKIP:6983 size=4348 bytes
x264 [debug]: frame= 865 QP=20.66 NAL=0 Slice:B Poc:28 I:351 P:2526 SKIP:5209 size=18796 bytes
x264 [debug]: frame= 866 QP=20.27 NAL=0 Slice:B Poc:30 I:8 P:1186 SKIP:6949 size=4796 bytes
x264 [debug]: frame= 867 QP=18.46 NAL=2 Slice:P Poc:34 I:106 P:6314 SKIP:1740 size=87418 bytes
x264 [debug]: frame= 868 QP=19.33 NAL=2 Slice:P Poc:36 I:11 P:3529 SKIP:4620 size=27036 bytes
x264 [debug]: frame= 869 QP=19.05 NAL=2 Slice:P Poc:42 I:364 P:6160 SKIP:1636 size=108525 bytes
x264 [debug]: frame= 870 QP=21.45 NAL=0 Slice:B Poc:38 I:312 P:1720 SKIP:6073 size=11862 bytes
x264 [debug]: frame= 871 QP=21.16 NAL=0 Slice:B Poc:40 I:2 P:1380 SKIP:6751 size=4457 bytes
x264 [debug]: frame= 872 QP=18.46 NAL=2 Slice:P Poc:44 I:127 P:6484 SKIP:1549 size=122050 bytes
x264 [debug]: frame= 873 QP=19.81 NAL=2 Slice:P Poc:46 I:33 P:3794 SKIP:4333 size=33795 bytes
x264 [debug]: frame= 874 QP=20.14 NAL=2 Slice:P Poc:48 I:84 P:6047 SKIP:2029 size=56662 bytes
x264 [debug]: frame= 875 QP=18.18 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=351437 bytes
x264 [debug]: frame= 876 QP=19.38 NAL=2 Slice:P Poc:2 I:181 P:3686 SKIP:4293 size=42702 bytes
x264 [debug]: frame= 877 QP=19.54 NAL=2 Slice:P Poc:4 I:78 P:6400 SKIP:1682 size=77160 bytes
x264 [debug]: frame= 878 QP=19.45 NAL=2 Slice:P Poc:12 I:210 P:6360 SKIP:1590 size=106371 bytes
x264 [debug]: frame= 879 QP=21.04 NAL=0 Slice:B Poc:6 I:13 P:2263 SKIP:5815 size=9389 bytes
x264 [debug]: frame= 880 QP=21.07 NAL=0 Slice:B Poc:8 I:329 P:4414 SKIP:3171 size=35889 bytes
x264 [debug]: frame= 881 QP=20.94 NAL=0 Slice:B Poc:10 I:10 P:2015 SKIP:6107 size=8771 bytes
x264 [debug]: frame= 882 QP=19.37 NAL=2 Slice:P Poc:14 I:54 P:6370 SKIP:1736 size=85357 bytes
x264 [debug]: frame= 883 QP=21.15 NAL=2 Slice:P Poc:22 I:51 P:5336 SKIP:2773 size=26726 bytes
x264 [debug]: frame= 884 QP=24.31 NAL=0 Slice:B Poc:16 I:3 P:1063 SKIP:7067 size=4072 bytes
x264 [debug]: frame= 885 QP=24.86 NAL=0 Slice:B Poc:18 I:54 P:1180 SKIP:6897 size=4715 bytes
x264 [debug]: frame= 886 QP=22.85 NAL=0 Slice:B Poc:20 I:8 P:112 SKIP:8039 size=363 bytes
x264 [debug]: frame= 887 QP=19.33 NAL=2 Slice:P Poc:24 I:157 P:6309 SKIP:1694 size=87289 bytes
x264 [debug]: frame= 888 QP=21.37 NAL=2 Slice:P Poc:32 I:60 P:4387 SKIP:3713 size=25172 bytes
x264 [debug]: frame= 889 QP=25.43 NAL=0 Slice:B Poc:26 I:2 P:584 SKIP:7564 size=2093 bytes
x264 [debug]: frame= 890 QP=25.55 NAL=0 Slice:B Poc:28 I:54 P:787 SKIP:7311 size=2858 bytes
x264 [debug]: frame= 891 QP=21.73 NAL=0 Slice:B Poc:30 I:3 P:108 SKIP:8049 size=320 bytes
x264 [debug]: frame= 892 QP=19.54 NAL=2 Slice:P Poc:34 I:176 P:6186 SKIP:1798 size=82878 bytes
x264 [debug]: frame= 893 QP=19.38 NAL=2 Slice:P Poc:42 I:276 P:6092 SKIP:1792 size=80715 bytes
x264 [debug]: frame= 894 QP=20.90 NAL=0 Slice:B Poc:36 I:6 P:1634 SKIP:6457 size=6903 bytes
x264 [debug]: frame= 895 QP=21.04 NAL=0 Slice:B Poc:38 I:276 P:3546 SKIP:4229 size=19856 bytes
x264 [debug]: frame= 896 QP=20.75 NAL=0 Slice:B Poc:40 I:10 P:1643 SKIP:6469 size=6855 bytes
x264 [debug]: frame= 897 QP=19.40 NAL=2 Slice:P Poc:44 I:81 P:6042 SKIP:2037 size=60195 bytes
x264 [debug]: frame= 898 QP=18.39 NAL=2 Slice:P Poc:46 I:13 P:5619 SKIP:2528 size=54585 bytes
x264 [debug]: frame= 899 QP=19.75 NAL=2 Slice:P Poc:48 I:71 P:5221 SKIP:2868 size=44168 bytes
x264 [debug]: frame= 900 QP=17.43 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=383698 bytes
x264 [debug]: frame= 901 QP=19.50 NAL=2 Slice:P Poc:4 I:388 P:6060 SKIP:1712 size=84226 bytes
x264 [debug]: frame= 902 QP=21.17 NAL=0 Slice:B Poc:2 I:7 P:2383 SKIP:5737 size=9569 bytes
x264 [debug]: frame= 903 QP=19.43 NAL=2 Slice:P Poc:12 I:261 P:6384 SKIP:1515 size=99410 bytes
x264 [debug]: frame= 904 QP=21.45 NAL=0 Slice:B Poc:6 I:6 P:2162 SKIP:5935 size=9493 bytes
x264 [debug]: frame= 905 QP=21.61 NAL=0 Slice:B Poc:8 I:300 P:4073 SKIP:3650 size=28484 bytes
x264 [debug]: frame= 906 QP=21.30 NAL=0 Slice:B Poc:10 I:4 P:1868 SKIP:6265 size=7808 bytes
x264 [debug]: frame= 907 QP=19.46 NAL=2 Slice:P Poc:14 I:124 P:6140 SKIP:1896 size=67776 bytes
x264 [debug]: frame= 908 QP=22.53 NAL=2 Slice:P Poc:22 I:82 P:3941 SKIP:4137 size=16988 bytes
x264 [debug]: frame= 909 QP=24.61 NAL=0 Slice:B Poc:16 I:3 P:173 SKIP:7984 size=472 bytes
x264 [debug]: frame= 910 QP=26.42 NAL=0 Slice:B Poc:18 I:24 P:186 SKIP:7946 size=767 bytes
x264 [debug]: frame= 911 QP=28.00 NAL=0 Slice:B Poc:20 I:0 P:2 SKIP:8158 size=44 bytes
x264 [debug]: frame= 912 QP=19.44 NAL=2 Slice:P Poc:24 I:161 P:6300 SKIP:1699 size=107741 bytes
x264 [debug]: frame= 913 QP=19.37 NAL=2 Slice:P Poc:32 I:113 P:6248 SKIP:1799 size=78931 bytes
x264 [debug]: frame= 914 QP=21.20 NAL=0 Slice:B Poc:26 I:9 P:1389 SKIP:6726 size=5601 bytes
x264 [debug]: frame= 915 QP=21.45 NAL=0 Slice:B Poc:28 I:309 P:3122 SKIP:4582 size=21757 bytes
x264 [debug]: frame= 916 QP=20.69 NAL=0 Slice:B Poc:30 I:18 P:1392 SKIP:6728 size=6088 bytes
x264 [debug]: frame= 917 QP=19.37 NAL=2 Slice:P Poc:34 I:57 P:5959 SKIP:2144 size=62767 bytes
x264 [debug]: frame= 918 QP=21.90 NAL=2 Slice:P Poc:42 I:53 P:6123 SKIP:1984 size=44854 bytes
x264 [debug]: frame= 919 QP=22.11 NAL=0 Slice:B Poc:36 I:0 P:436 SKIP:7720 size=1131 bytes
x264 [debug]: frame= 920 QP=23.59 NAL=0 Slice:B Poc:38 I:145 P:1094 SKIP:6890 size=5104 bytes
x264 [debug]: frame= 921 QP=24.24 NAL=0 Slice:B Poc:40 I:39 P:553 SKIP:7557 size=1764 bytes
x264 [debug]: frame= 922 QP=18.48 NAL=2 Slice:P Poc:44 I:223 P:6311 SKIP:1626 size=121440 bytes
x264 [debug]: frame= 923 QP=18.45 NAL=2 Slice:P Poc:46 I:18 P:4151 SKIP:3991 size=37080 bytes
x264 [debug]: frame= 924 QP=20.29 NAL=2 Slice:P Poc:48 I:29 P:3770 SKIP:4361 size=19268 bytes
x264 [debug]: frame= 925 QP=17.44 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=393041 bytes
x264 [debug]: frame= 926 QP=19.41 NAL=2 Slice:P Poc:2 I:67 P:3334 SKIP:4759 size=33998 bytes
x264 [debug]: frame= 927 QP=18.43 NAL=2 Slice:P Poc:4 I:106 P:6399 SKIP:1655 size=93919 bytes
x264 [debug]: frame= 928 QP=22.04 NAL=2 Slice:P Poc:10 I:81 P:3940 SKIP:4139 size=19024 bytes
x264 [debug]: frame= 929 QP=24.64 NAL=0 Slice:B Poc:6 I:1 P:260 SKIP:7895 size=721 bytes
x264 [debug]: frame= 930 QP=24.83 NAL=0 Slice:B Poc:8 I:42 P:246 SKIP:7871 size=985 bytes
x264 [debug]: frame= 931 QP=19.38 NAL=2 Slice:P Poc:16 I:265 P:6285 SKIP:1610 size=98410 bytes
x264 [debug]: frame= 932 QP=25.75 NAL=0 Slice:B Poc:12 I:21 P:489 SKIP:7649 size=1509 bytes
x264 [debug]: frame= 933 QP=24.72 NAL=0 Slice:B Poc:14 I:38 P:552 SKIP:7557 size=2448 bytes
x264 [debug]: frame= 934 QP=18.47 NAL=2 Slice:P Poc:22 I:328 P:6290 SKIP:1542 size=118060 bytes
x264 [debug]: frame= 935 QP=20.90 NAL=0 Slice:B Poc:18 I:255 P:3410 SKIP:4296 size=27900 bytes
x264 [debug]: frame= 936 QP=20.83 NAL=0 Slice:B Poc:20 I:4 P:1513 SKIP:6611 size=5985 bytes
x264 [debug]: frame= 937 QP=19.38 NAL=2 Slice:P Poc:24 I:69 P:6102 SKIP:1989 size=63541 bytes
x264 [debug]: frame= 938 QP=21.84 NAL=2 Slice:P Poc:32 I:59 P:5413 SKIP:2688 size=32897 bytes
x264 [debug]: frame= 939 QP=23.90 NAL=0 Slice:B Poc:26 I:5 P:550 SKIP:7603 size=1503 bytes
x264 [debug]: frame= 940 QP=23.22 NAL=0 Slice:B Poc:28 I:89 P:577 SKIP:7491 size=2636 bytes
x264 [debug]: frame= 941 QP=24.20 NAL=0 Slice:B Poc:30 I:26 P:375 SKIP:7757 size=1391 bytes
x264 [debug]: frame= 942 QP=18.40 NAL=2 Slice:P Poc:34 I:242 P:6380 SKIP:1538 size=127254 bytes
x264 [debug]: frame= 943 QP=20.59 NAL=2 Slice:P Poc:44 I:61 P:6091 SKIP:2008 size=54571 bytes
x264 [debug]: frame= 944 QP=23.49 NAL=0 Slice:B Poc:36 I:4 P:1826 SKIP:6263 size=7509 bytes
x264 [debug]: frame= 945 QP=22.70 NAL=0 Slice:B Poc:38 I:198 P:2360 SKIP:5528 size=11225 bytes
x264 [debug]: frame= 946 QP=22.36 NAL=0 Slice:B Poc:40 I:192 P:825 SKIP:7113 size=5411 bytes
x264 [debug]: frame= 947 QP=23.56 NAL=0 Slice:B Poc:42 I:131 P:1886 SKIP:6062 size=8779 bytes
x264 [debug]: frame= 948 QP=18.45 NAL=2 Slice:P Poc:46 I:117 P:5919 SKIP:2124 size=85074 bytes
x264 [debug]: frame= 949 QP=18.40 NAL=2 Slice:P Poc:48 I:185 P:6196 SKIP:1779 size=88619 bytes
x264 [debug]: frame= 950 QP=17.43 NAL=3 Slice:I Poc:0 I:8160 P:0 SKIP:0 size=390893 bytes
x264 [debug]: frame= 951 QP=19.14 NAL=2 Slice:P Poc:2 I:160 P:3051 SKIP:4949 size=27402 bytes

[Guest]

My verifier is underflowing because I asked for 10Mbps and it is giving me 12Mbps:

encoded 1000 frames, 11.71 fps, 12103.11 kb/s

Now I need to see if this is serious and if it can be mitigated, possibly via ratetol and/or qcomp.

[Manao]

but still, the scenecut should be an I picture. You have black borders, which can be encoded as skip in P frames, thus x264 finds it better to encode the scene cut as a P frame.

[Guest]

Ah, OK. Still for seeking and authoring, it might be better to have an IDR at a hard cut.

Now I'll start a full movie two-pass encode and pray that x264 doesn't report underflow.

[Dark Shikari]

Wouldn't help, as the big picture is not being rendered as an I picture, as it should be.Scenecut detection isn't guaranteed to make all scenecuts I-frames; its strength is weighted by the distance to the next keyframe.

VBV should work correctly with or without scenecut detection anyways--it shouldn't rely on it.My verifier is underflowing because I asked for 10Mbps and it is giving me 12Mbps:

encoded 1000 frames, 11.71 fps, 12103.11 kb/s

Now I need to see if this is serious and if it can be mitigated, possibly via ratetol and/or qcomp.Is x264 reporting any underflows? If x264 is producing a stream that underflows but isn't warning about it, this is critical bug and must be fixed. Don't "try to mitigate it with ratetol and qcomp"--help get it fixed!

Akupenguin has a script that takes an input passfile (from the pass you just ran) and tests it to see if there was an underflow, in case x264 was wrong.

[Guest]

Scenecut detection isn't guaranteed to make all scenecuts I-frames; its strength is weighted by the distance to the next keyframe. OK, it's not a big deal for me as I don't really need it to be an I picture.

VBV should work correctly with or without scenecut detection anyways--it shouldn't rely on it.Is x264 reporting any underflows? I'll tell you after my full two-pass encode finishes. That will be tomorrow about this time. :)

If x264 is producing a stream that underflows but isn't warning about it, this is critical bug and must be fixed. Don't "try to mitigate it with ratetol and qcomp"--help get it fixed! OK, sure.

Akupenguin has a script that takes an input passfile (from the pass you just ran) and tests it to see if there was an underflow, in case x264 was wrong. Can I have that script? :)

[Guest]

One more question, Dark.

If x264 reports an underflow, isn't that a bug in and of itself? I mean, how is a user supposed to fix that if it happens?

[Dark Shikari]

Can I have that script? :)Linkage (http://akuvian.org/src/x264/vbv.pl).One more question, Dark.

If x264 reports an underflow, isn't that a bug in and of itself? I mean, how is a user supposed to fix that if it happens?There are three types of VBV underflow bugs (ranging from most important to least):

1. x264 underflows, but fails to print a warning. This means the internal VBV model is wrong--a critical problem.

2. x264 underflows and prints a warning, but underflows in a case that should be trivially avoidable even without lookahead.

3. x264 underflows and prints a warning, but underflows in a case that is not easily avoidable.

We can't do anything about the third case without adding VBV emergency mode.

[Guest]

2. x264 underflows and prints a warning, but underflows in a case that should be trivially avoidable even without lookahead. You mean during the second pass it is corrected?

Thanks for the link.

[Dark Shikari]

You mean during the second pass it is corrected?A 2)-type problem might or might not be corrected in the second pass, depending on what the problem actually was.

Note that all of these are compliance problems, not quality problems (the latter of which have their own hierarchy).

[Guest]

Note that all of these are compliance problems, not quality problems (the latter of which have their own hierarchy). But if an underflow occurs, the video will freeze.

[Dark Shikari]

But if an underflow occurs, the video will freeze.Of course, the entire point being that a compliance problem is worse than a quality problem. A "quality problem" is defined as a case where VBV unnecessarily decreases quality in a section but is still compliant.

[Manao]

3. x264 underflows and prints a warning, but underflows in a case that is not easily avoidable.x264 has MBRC, so underflow can be avoided even without lookahead.

[Dark Shikari]

x264 has MBRC, so underflow can be avoided even without lookahead.I know, I was referring to the case where "QP51 isn't enough", or a case where a bug in MBRC causes that to fail.

[Guest]

We can't do anything about the third case without adding VBV emergency mode. How hard would that be, IYHO?

[Dark Shikari]

How hard would that be, IYHO?There are various ways to do it.

1. The way Mainconcept does it--after encoding a frame, if VBV was violated, replace it with a frame full of skips.

I don't like this way because it is very suboptimal for the case of "QP51 is barely enough" in which case just thresholding down more coefficients and using a higher lambda would have done better. It also fails on I-frames. It's easy though.

2. Allow QPs above 51 (clip to 51 on the actual quantization), and put if( vbv emergency ) denoise_dct calls everywhere. My thought was something like the following:

("denoise" means "decrease", so at QP70 there would be no residual left anywhere)

QP 55-60: denoise AC coefficients
QP 61-65: denoise DC coefficients of normal DCT and AC coefficients of i16x16/chroma DC.
QP 65-70: denoise DC coefficients of i16x16/chroma DC.
QP 71: Emergency mode, all blocks become skips in P-frames and empty-i16x16 in I-frames.

I have a partial patch for this lying around, but it's messy. The ideal would be to use trellis to lower coefficients: set it so that it always checks zero as a coefficient option and just keep raising its lambda at higher and higher QPs. But this would require trellis to do VBV emergency mode.

[Guest]

Thanks for that information, Dark.

Do you have a patch for the Main Concept way? It's OK for me as there are only a few underflows so I can accept suboptimality. What do you mean when you say it fails on I frames?

Also, on another topic. If I compile x264 in Visual Studio rather than MSYS, will I lose performance, possibly because of some ASM not being used?

[Dark Shikari]

Do you have a patch for the Main Concept way? It's OK for me as there are only a few underflows so I can accept suboptimality.Here's the patch I have (http://pastebin.com/m21c5265c) for the second method. No guarantee it'll patch, but from what I recall it did actually work, though it was not ideal. It doesn't work when --nr at the same time.
What do you mean when you say it fails on I frames?Mainconcept's algorithm only replaces P/B-frames, so I-frames still can cause underflows if they can't get small enough at QP51.
Also, on another topic. If I compile x264 in Visual Studio rather than MSYS, will I lose performance, possibly because of some ASM not being used?Yes. If you want to use another compiler, use ICC; Dopefish has a set of patches to add ICC support (posted I think in either current patches and/or win64 thread), including for the inline ASM x264 uses as well. This will give a bit better performance than GCC.

[Manao]

Just a reminder of the obvious, but "QP51 isn't enough" isn't the problem here, since I frame were encoded between QP20 and QP30. So that patch won't fix underflows.

[Inventive Software]

Just a reminder of the obvious, but "QP51 isn't enough" isn't the problem here, since I frame were encoded between QP20 and QP30. So that patch won't fix underflows.

The frame in question is a P-frame, not an I-frame.

Reading this thread, would it not be worthwhile having a bigger lookahead, possibly 2 GOPs or more?

[Dark Shikari]

The frame in question is a P-frame, not an I-frame.

Reading this thread, would it not be worthwhile having a bigger lookahead, possibly 2 GOPs or more?There is no lookahead at all for VBV currently. Or, more specifically, the lookahead is the size of a minigop--the next frame to be encoded, plus any related B-frames.

[Inventive Software]

There is no lookahead at all for VBV currently. Or, more specifically, the lookahead is the size of a minigop--the next frame to be encoded, plus any related B-frames.

Right, so this needs to be implemented then?! :devil:

I'd tackle it, but my coding leaves a lot to be desired at the moment. :rolleyes:

[Dark Shikari]

Right, so this needs to be implemented then?! :devil:

I'd tackle it, but my coding leaves a lot to be desired at the moment. :rolleyes:VBV lookahead can't really be implemented until we have separated the slicetype decision into a separate thread. Once that happens, doing VBV lookahead is actually rather trivial.

Avail has a somewhat-working patch for this, but it's still very alpha. Thanks to Kaz for his work on it so far.

[Inventive Software]

VBV lookahead can't really be implemented until we have separated the slicetype decision into a separate thread. Once that happens, doing VBV lookahead is actually rather trivial.

Avail has a somewhat-working patch for this, but it's still very alpha. Thanks to Kaz for his work on it so far.

OK, reading that makes it sound like there could be a solution. Is there a way to hack something very crudely together for those needing immediate correct 1-pass CBR? Like using 1 thread for example? And then doing it properly later on, when some of the ground-work's been redone?