VBV Underflows in x264 (tested with 1281)

[Blue_MiSfit]

Hey devs,

I've got some VBV bugs to report :(

I'm encoding the trailer for "The Prestige" from Apple's trailer park.

http://www.apple.com/trailers/touchstone/theprestige/

I downloaded the 1080p version, remuxed to MKV, and downscaled it to 480p, with mattes to display properly at 16x9 AR, like this:


dss2("the_prestige_h1080p.mkv")

degrainmedian

spline36resize(720,360)
addborders(0,60,0,60)


When I encode with x264 rev 1281 using the following settings, I get bad results (lots of underflow errors)


x264 "prestige.avs" --pass 1 --bitrate 1500 --vbv-maxr
ate 2000 --vbv-bufsize 2000 --preset slower --tune film --output NUL --stats ".s
tats" --thread-input
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 3.0
x264 [info]: frame I:85 Avg QP:13.38 size: 29383
x264 [info]: frame P:1713 Avg QP:14.78 size: 9001
x264 [info]: frame B:1899 Avg QP:15.54 size: 4841
x264 [info]: consecutive B-frames: 14.6% 32.7% 40.0% 12.7%
x264 [info]: mb I I16..4: 46.4% 0.0% 53.6%
x264 [info]: mb P I16..4: 24.8% 0.0% 0.0% P16..4: 30.8% 0.0% 0.0% 0.0% 0
.0% skip:44.3%
x264 [info]: mb B I16..4: 5.5% 0.0% 0.0% B16..8: 24.6% 0.0% 0.0% direct:
19.8% skip:50.1% L0:33.3% L1:43.0% BI:23.7%
x264 [info]: direct mvs spatial:97.3% temporal:2.7%
x264 [info]: coded y,uvDC,uvAC intra: 58.7% 52.5% 34.3% inter: 20.4% 14.6% 1.8%
x264 [info]: i16 v,h,dc,p: 49% 20% 18% 13%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 16% 16% 6% 7% 8% 6% 8% 6%
x264 [info]: kb/s:1406.41

encoded 3697 frames, 21.61 fps, 1406.41 kb/s

x264 "prestige.avs" --pass 2 --bitrate 2000 --vbv-maxr
ate 2000 --vbv-bufsize 2000 --preset slower --tune film --output "Prestige_2mbps
_1281.mp4" --stats ".stats" --thread-input
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [warning]: target: 2000.00 kbit/s, expected: 1871.24 kbit/s, avg QP: 16.778
1
x264 [info]: profile High, level 3.1
mp4 [info]: initial delay 417083 (scale 10000000)
x264 [warning]: VBV underflow (frame 664, -54889 bits)0:03:41
x264 [warning]: VBV underflow (frame 665, -76199 bits)
x264 [warning]: VBV underflow (frame 666, -59479 bits)0:03:41
x264 [warning]: VBV underflow (frame 1284, -17735 bits)0:03:13
x264 [warning]: VBV underflow (frame 1296, -59497 bits)0:03:12
x264 [warning]: VBV underflow (frame 1498, -56224 bits)0:02:57
x264 [warning]: VBV underflow (frame 1500, -18039 bits)0:02:57
x264 [warning]: VBV underflow (frame 1501, -2423 bits)
x264 [warning]: VBV underflow (frame 1931, -41900 bits)0:02:20
x264 [warning]: VBV underflow (frame 2422, -4282 bits) 0:01:39
x264 [warning]: VBV underflow (frame 2508, -97581 bits)0:01:32
x264 [warning]: VBV underflow (frame 2509, -152007 bits)
x264 [warning]: VBV underflow (frame 2510, -157895 bits)
x264 [warning]: VBV underflow (frame 2526, -57998 bits)0:01:31
x264 [warning]: VBV underflow (frame 2527, -98359 bits)
x264 [warning]: VBV underflow (frame 2529, -35239 bits)0:01:31
x264 [warning]: VBV underflow (frame 2567, -21353 bits)0:01:28
x264 [warning]: VBV underflow (frame 2568, -75847 bits)0:01:28
x264 [warning]: VBV underflow (frame 2569, -95687 bits)
x264 [warning]: VBV underflow (frame 2570, -42863 bits)
x264 [warning]: VBV underflow (frame 2572, -37367 bits)0:01:27
x264 [warning]: VBV underflow (frame 2580, -9171 bits) 0:01:27
x264 [warning]: VBV underflow (frame 2840, -39213 bits)0:01:07
x264 [warning]: VBV underflow (frame 2841, -5415 bits) 0:01:07
x264 [warning]: VBV underflow (frame 3084, -37863 bits)0:00:48
x264 [warning]: VBV underflow (frame 3085, -37119 bits)
x264 [warning]: VBV underflow (frame 3086, -37743 bits)
x264 [warning]: VBV underflow (frame 3087, -46279 bits)0:00:48
x264 [info]: frame I:85 Avg QP:13.41 size: 35940
x264 [info]: frame P:1713 Avg QP:13.31 size: 11742
x264 [info]: frame B:1899 Avg QP:14.41 size: 6120
x264 [info]: consecutive B-frames: 14.6% 32.7% 40.0% 12.7%
x264 [info]: mb I I16..4: 33.0% 42.1% 24.9%
x264 [info]: mb P I16..4: 4.5% 10.2% 3.7% P16..4: 17.3% 10.0% 8.3% 0.6% 0
.7% skip:44.8%
x264 [info]: mb B I16..4: 0.8% 2.5% 1.0% B16..8: 24.7% 2.3% 4.0% direct:
10.1% skip:54.6% L0:36.3% L1:42.3% BI:21.4%
x264 [info]: 8x8 transform intra:53.4% inter:33.3%
x264 [info]: direct mvs spatial:82.0% temporal:18.0%
x264 [info]: coded y,uvDC,uvAC intra: 70.8% 70.1% 63.5% inter: 21.6% 15.7% 6.5%
x264 [info]: i16 v,h,dc,p: 57% 18% 11% 13%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 11% 19% 7% 8% 10% 8% 10% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 9% 14% 8% 11% 11% 9% 10% 12%
x264 [info]: ref P L0: 74.5% 12.4% 5.9% 2.5% 1.7% 1.3% 1.1% 0.7%
x264 [info]: ref B L0: 85.4% 7.4% 3.3% 1.5% 1.0% 0.7% 0.6%
x264 [info]: kb/s:1805.03

encoded 3697 frames, 13.11 fps, 1805.03 kb/s


I know my settings are a little strange, but I frequently re-use a firstpass stats file to encode the same source at multiple bitrates.

I've tried this with an older version, and the results were visually much worse. So, it seems some of the VBV improvements made over the last few months help, but don't eliminate this problem.

I also tried encoding this source in pure CBR, and got the following results:

x264 "prestige.avs" --pass 1 --bitrate 2000 --vbv-maxr
ate 2000 --vbv-bufsize 2000 --preset slower --tune film --output NUL --stats ".s
tats" --thread-input
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 3.0
x264 [info]: frame I:85 Avg QP:13.10 size: 30778
x264 [info]: frame P:1713 Avg QP:13.44 size: 10282
x264 [info]: frame B:1899 Avg QP:14.44 size: 5936
x264 [info]: consecutive B-frames: 14.6% 32.7% 40.0% 12.7%
x264 [info]: mb I I16..4: 46.2% 0.0% 53.8%
x264 [info]: mb P I16..4: 25.2% 0.0% 0.0% P16..4: 31.2% 0.0% 0.0% 0.0% 0
.0% skip:43.5%
x264 [info]: mb B I16..4: 5.8% 0.0% 0.0% B16..8: 25.4% 0.0% 0.0% direct:
21.9% skip:46.8% L0:32.6% L1:42.6% BI:24.7%
x264 [info]: direct mvs spatial:97.1% temporal:2.9%
x264 [info]: coded y,uvDC,uvAC intra: 61.0% 53.9% 37.5% inter: 23.7% 16.1% 2.8%
x264 [info]: i16 v,h,dc,p: 48% 20% 19% 14%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 16% 16% 6% 7% 8% 6% 8% 6%
x264 [info]: kb/s:1634.36

encoded 3697 frames, 27.55 fps, 1634.36 kb/s


x264 "prestige.avs" --pass 2 --bitrate 2000 --vbv-maxr
ate 2000 --vbv-bufsize 2000 --preset slower --tune film --output "prestige_2mbps
_2passCBR.mp4" --stats ".stats" --thread-input
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [warning]: target: 2000.00 kbit/s, expected: 1803.02 kbit/s, avg QP: 16.697
1
x264 [info]: profile High, level 3.1
mp4 [info]: initial delay 417083 (scale 10000000)
x264 [warning]: VBV underflow (frame 664, -51289 bits)0:03:34
x264 [warning]: VBV underflow (frame 665, -117575 bits)
x264 [warning]: VBV underflow (frame 666, -29375 bits)0:03:34
x264 [warning]: VBV underflow (frame 1284, -53639 bits)0:03:00
x264 [warning]: VBV underflow (frame 1498, -52968 bits)0:02:43
x264 [warning]: VBV underflow (frame 1500, -12767 bits)0:02:43
x264 [warning]: VBV underflow (frame 1923, -44953 bits)0:02:13
x264 [warning]: VBV underflow (frame 1924, -36871 bits)
x264 [warning]: VBV underflow (frame 1925, -6119 bits)
x264 [warning]: VBV underflow (frame 1931, -4740 bits) 0:02:12
x264 [warning]: VBV underflow (frame 2274, -59912 bits)0:01:46
x264 [warning]: VBV underflow (frame 2508, -142551 bits):01:28
x264 [warning]: VBV underflow (frame 2509, -149047 bits)
x264 [warning]: VBV underflow (frame 2510, -145367 bits)
x264 [warning]: VBV underflow (frame 2526, -36454 bits)0:01:27
x264 [warning]: VBV underflow (frame 2527, -96687 bits)
x264 [warning]: VBV underflow (frame 2529, -36887 bits)0:01:27
x264 [warning]: VBV underflow (frame 2568, -46937 bits)0:01:24
x264 [warning]: VBV underflow (frame 2569, -84703 bits)
x264 [warning]: VBV underflow (frame 2570, -40687 bits)
x264 [warning]: VBV underflow (frame 2572, -13303 bits)0:01:24
x264 [warning]: VBV underflow (frame 2839, -160336 bits):01:04
x264 [warning]: VBV underflow (frame 2840, -37847 bits)
x264 [warning]: VBV underflow (frame 3087, -20532 bits)0:00:46
x264 [warning]: VBV underflow (frame 3107, -1908 bits) 0:00:45
x264 [info]: frame I:85 Avg QP:13.24 size: 36950
x264 [info]: frame P:1713 Avg QP:13.48 size: 11690
x264 [info]: frame B:1899 Avg QP:14.52 size: 5995
x264 [info]: consecutive B-frames: 14.6% 32.7% 40.0% 12.7%
x264 [info]: mb I I16..4: 32.5% 42.0% 25.5%
x264 [info]: mb P I16..4: 4.5% 10.4% 3.7% P16..4: 17.0% 10.0% 8.4% 0.6% 0
.7% skip:44.9%
x264 [info]: mb B I16..4: 0.8% 2.5% 1.0% B16..8: 25.0% 2.3% 4.1% direct:
9.8% skip:54.4% L0:35.9% L1:42.1% BI:22.0%
x264 [info]: 8x8 transform intra:53.9% inter:33.3%
x264 [info]: direct mvs spatial:78.0% temporal:22.0%
x264 [info]: coded y,uvDC,uvAC intra: 71.2% 70.4% 64.1% inter: 21.4% 15.5% 6.4%
x264 [info]: i16 v,h,dc,p: 56% 18% 12% 14%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 11% 19% 7% 8% 10% 8% 10% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 9% 14% 8% 11% 11% 9% 10% 12%
x264 [info]: ref P L0: 74.5% 12.4% 5.9% 2.6% 1.7% 1.3% 1.1% 0.7%
x264 [info]: ref B L0: 85.5% 7.4% 3.2% 1.5% 1.0% 0.7% 0.6%
x264 [info]: kb/s:1792.46

encoded 3697 frames, 13.53 fps, 1792.46 kb/s


~MiSfit

[Dark Shikari]

I know my settings are a little strange, but I frequently re-use a firstpass stats file to encode the same source at multiple bitrates.Atrocious idea if you care at all about VBV compliance.

[Blue_MiSfit]

Granted.

But I got the same results with a standard 2 pass CBR encode.

~MiSfit

[Dark Shikari]

Granted.

But I got the same results with a standard 2 pass CBR encode.

~MiSfitVBV has known issues with Gabriel's 2-pass algorithm. My 1-pass algorithm works fine, and furthermore, gives better results than 2-pass does (in CBR, at least).

Avoid 2-pass if it's causing you issues.

[ACrowley]

Same Problem here as i said in the other Thread. 2 pass VBR with 40Mbit VBV Maxrate. 1 pass works fine.
But it looks like i get only VBV underflows with the Tune-Grain options. Everything works fine without Tune Grain.
And-Or its caused by the excessive Grain in some Scenes of XMen Origins Wolferine BluRay Source.
I can see that the Bitrate rises "very high" Peakes in those Scenes

CRF 1 Pass Mode works fine on this Source

[Dark Shikari]

Have you tried --slow-firstpass to see if the problems go away?

If so, it's a problem with inaccurate bitrate prediction on the first pass and 2-pass VBV's inability to properly adapt.

[Boolsheet]

Have you tried --slow-firstpass to see if the problems go away?
Oh ok, looks like you found something. I tested the Island trailer again (like we did over here (http://forum.doom9.org/showthread.php?t=108571&page=6)). Without first-slowpass it still gives underflow warnings on the second pass. With slow-firstpass it underflows in the first pass, the second completes without warnings.

[Dark Shikari]

Oh ok, looks like you found something. I tested the Island trailer again (like we did over here (http://forum.doom9.org/showthread.php?t=108571&page=6)). Without first-slowpass it still gives underflow warnings on the second pass. With slow-firstpass it underflows in the first pass, the second completes without warnings.I can't find the link to the original Island source--if you can give me it and your exact settings, I can investigate.

1-pass VBV issues are prioritized over 2-pass.

[Boolsheet]

Sure, both passes used the same:
x264.1281.x64.x264nl.exe -p 1 --bitrate 40000 --vbv-maxrate 40000 --vbv-bufsize 30000 --slow-firstpass -o NUL TheIsland.avs
x264.1281.x64.x264nl.exe -p 2 --bitrate 40000 --vbv-maxrate 40000 --vbv-bufsize 30000 --slow-firstpass -o NUL TheIsland.avs

The full trailer is still available on bens SkyDrive (http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/The%20Island%20Trailer).
This is the part with the last 419 frames Revgen uploaded: http://www.megaupload.com/?d=RM8TF0R0. I'm sure this will underflow too, but I did it over the full trailer.

[Dark Shikari]

How many threads are being used?

[Boolsheet]

It's on auto, so 12.

Here's a Island Trailer mirror: http://shimapan.users.sourceforge.net/island/ .

[Blue_MiSfit]

@DS:

So, to use 1-pass VBV mode, I simply encode only once? No "--pass x" parameter?

~MiSfit

[Dark Shikari]

@DS:

So, to use 1-pass VBV mode, I simply encode only once? No "--pass x" parameter?

~MiSfitYes... encoding in one pass means encoding in one pass...

[Audionut]

Wouldn't omitting --pass imply default --crf, or does -B come into affect?

[ACrowley]

Oh, i get massive VBV Underflows in 1 pass CRF Mode too with this Source!
A 1 pass CRF 18 with VBV 30/40 Mbit. So it looks like CRF 1 pass outputs underflows too!


Source: F:\grain.avs
Output: F:\encode.mkv
Preset: Veryslow
Tuning: Grain
Profile: High
Params: --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

Analayzing source file:
F:\grain.avs: 1920x816, 48000/2002 fps, 1782 frames

Resolution: 1920 x 816
Frame No. : 1782
Frame Rate: 24000/1001

x264 0.77.1281M 39f6499
built by techouse on Oct 8 2009, gcc: 4.4.1 (x86_64.core2.Komisar)
Revision: 1281

Commandline for x264:
"C:\AV Tools\x264 Launcher\pipebuf.exe" "C:\AV Tools\x264 Launcher\avs2yuv.exe" F:\grain.avs -raw - : "C:\AV Tools\x264 Launcher\x264_x64.exe" --preset veryslow --tune grain --crf 18.0 --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3 --output F:\encode.mkv --frames 1782 --fps 24000/1001 - 1920x816 : 4

x264 : 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [warning]: VBV underflow (frame 1340, -695124 bits)
x264 [warning]: VBV underflow (frame 1341, -927371 bits)
x264 [warning]: VBV underflow (frame 1342, -380707 bits)
x264 [warning]: VBV underflow (frame 1343, -690475 bits)
x264 [warning]: VBV underflow (frame 1397, -816346 bits)
x264 [warning]: VBV underflow (frame 1398, -639491 bits)
x264 [warning]: VBV underflow (frame 1399, -530731 bits)
x264 [warning]: VBV underflow (frame 1400, -820275 bits)
x264 [warning]: VBV underflow (frame 1402, -738453 bits)
x264 [warning]: VBV underflow (frame 1403, -654243 bits)
x264 [warning]: VBV underflow (frame 1404, -639259 bits)
x264 [warning]: VBV underflow (frame 1405, -746043 bits)
x264 [warning]: VBV underflow (frame 1406, -836539 bits)
x264 [warning]: VBV underflow (frame 1558, -386799 bits)
x264 [warning]: VBV underflow (frame 1560, -721301 bits)
x264 [warning]: VBV underflow (frame 1561, -119731 bits)
x264 [warning]: VBV underflow (frame 1562, -187819 bits)
x264 [warning]: VBV underflow (frame 1568, -162888 bits)
x264 [warning]: VBV underflow (frame 1569, -533915 bits)
x264 [warning]: VBV underflow (frame 1570, -351307 bits)
x264 [warning]: VBV underflow (frame 1571, -334555 bits)
x264 [warning]: VBV underflow (frame 1576, -590030 bits)
x264 [warning]: VBV underflow (frame 1577, -1248691 bits)
x264 [warning]: VBV underflow (frame 1665, -487289 bits)
x264 [warning]: VBV underflow (frame 1666, -1285987 bits)
x264 [warning]: VBV underflow (frame 1667, -707811 bits)
x264 [warning]: VBV underflow (frame 1668, -1186171 bits)
x264 [warning]: VBV underflow (frame 1670, -1055893 bits)
x264 [warning]: VBV underflow (frame 1671, -426819 bits)
x264 [warning]: VBV underflow (frame 1672, -505371 bits)
x264 [warning]: VBV underflow (frame 1674, -375701 bits)
x264 [warning]: VBV underflow (frame 1682, -607622 bits)
x264 [warning]: VBV underflow (frame 1683, -551795 bits)
x264 [warning]: VBV underflow (frame 1684, -743131 bits)
x264 [warning]: VBV underflow (frame 1685, -343635 bits)
x264 [warning]: VBV underflow (frame 1686, -177467 bits)
x264 [warning]: VBV underflow (frame 1687, -351139 bits)
x264 [warning]: VBV underflow (frame 1688, -802547 bits)
x264 [warning]: VBV underflow (frame 1689, -581627 bits)
F:\grain.avs: 1920x816, 48000/2002 fps, 1782 frames

x264 [info]: frame I:131 Avg QP:19.95 size:174030
x264 [info]: frame P:594 Avg QP:20.31 size:174089
x264 [info]: frame B:1057 Avg QP:20.76 size:155463
x264 [info]: consecutive B-frames: 5.3% 17.9% 30.5% 46.3%
x264 [info]: mb I I16..4: 42.1% 48.5% 9.4%
x264 [info]: mb P I16..4: 17.7% 26.0% 9.0% P16..4: 26.7% 11.6% 6.7% 0.0% 0.0% skip: 2.3%
x264 [info]: mb B I16..4: 9.3% 10.9% 6.1% B16..8: 32.9% 2.8% 2.4% direct:20.1% skip:15.7% L0:35.0% L1:34.2% BI:30.8%
x264 [info]: 8x8 transform intra:46.2% inter:37.9%
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: coded y,uvDC,uvAC intra: 94.7% 85.5% 76.3% inter: 60.2% 44.5% 21.2%
x264 [info]: i16 v,h,dc,p: 7% 4% 48% 42%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 7% 16% 10% 12% 10% 11% 11% 14%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 7% 11% 11% 13% 11% 12% 12% 18%
x264 [info]: ref P L0: 62.3% 17.9% 10.5% 5.2% 4.1%
x264 [info]: ref B L0: 83.0% 9.8% 4.7% 2.5%
x264 [info]: kb/s:31271.74
encoded 1782 frames, 1.62 fps, 31271.74 kb/s

It depends on the Source with a lot of Grain
This one here was Xmen-Origins Wolferine Blruray, Frame 6957 -8737, the Scenes at the Beginning with excessive Grain.
So its impossible to encode this Source with rev1281 and my desired Settings at the Moment. I will try it without Tune Grain now

EDIT:
No,the same Underflows without Tune-Grain in 1 (CRF) and 2 pass Mode

EDIT2:

[I]No Underflows in 2 pass VBR Mode with --slow-firstpass!

[Dark Shikari]

Oh ok, looks like you found something. I tested the Island trailer again (like we did over here (http://forum.doom9.org/showthread.php?t=108571&page=6)). Without first-slowpass it still gives underflow warnings on the second pass. With slow-firstpass it underflows in the first pass, the second completes without warnings.Cannot replicate. No issues here:for pass in 1 2;do ./x264 i:\island.avs -o NUL --vbv-bufsize 30000 --vbv-maxrate 40000 --bitrate 40000 --pass $pass --slow-firstpass --threads 12;doneUntil someone can give me settings that produce the problem, I certainly can't do anything about it.This one here was Xmen-Origins Wolferine Blruray, Frame 6957 -8737, the Scenes at the Beginning with excessive Grain.I do not own a Blu-ray drive and I am not going to spend $500 just because people won't upload minimal test cases. If you don't upload short samples that I can use to replicate the problem, I am not going to bother trying to fix it.

[Dark Shikari]

Replicated the second pass problem with fast firstpass; it happened just as I expected. Still no luck with the supposed first pass problem.

[ACrowley]

Cannot replicate. No issues here:Until someone can give me settings that produce the problem, I certainly can't do anything about it.I do not own a Blu-ray drive and I am not going to spend $500 just because people won't upload minimal test cases. If you don't upload short samples that I can use to replicate the problem, I am not going to bother trying to fix it.

I will upload a sample

But at the moment i can find the right sample to reproduce the Problem.
Is it possible that the underflow doesnt occur everytime at the same Point ? Strange...

[Dark Shikari]

I will upload a sample

But at the moment i can find the right sample to reproduce the Problem.
Is it possible that the underflow doesnt occur everytime at the same Point ? Strange...Yes. VBV analysis is nondeterministic.

[ACrowley]

Yes. VBV analysis is nondeterministic.

Ah that explains why i cant reproduce VBV underflows at the Moment with the the same Sample+264 cmdline where i had those massive underflows in 1 and 2 pass Mode earlier today

EDIT
Ah....now the underflows are back (1pass crf 18 :)
I will upload the sample

[Dark Shikari]

So, uh, guys, I have some spoilers for you.

These 2-pass issues? Yeah...

... they were caused by yet another bug inadvertently introduced in MB-tree.

We are without a doubt going to be fixing bugs in MB-tree until the end of time.

(Locally committed.)

[Blue_MiSfit]

:) Can't wait to see the results.

~MiSft

[ACrowley]

Here is the Sample from X-Men Origins Wolferine BluRay which contains a lot of grain and produce VBV Underflows in 1 Pass CRF and 2 Pass VBR
http://www.megaupload.com/?d=V86HB349

[Dark Shikari]

x264 [info]: frame I:131 Avg QP:19.95 size:174030
x264 [info]: frame P:594 Avg QP:20.31 size:174089
x264 [info]: frame B:1057 Avg QP:20.76 size:155463
x264 [info]: consecutive B-frames: 5.3% 17.9% 30.5% 46.3%
x264 [info]: mb I I16..4: 42.1% 48.5% 9.4%
x264 [info]: mb P I16..4: 17.7% 26.0% 9.0% P16..4: 26.7% 11.6% 6.7% 0.0% 0.0% skip: 2.3%
x264 [info]: mb B I16..4: 9.3% 10.9% 6.1% B16..8: 32.9% 2.8% 2.4% direct:20.1% skip:15.7% L0:35.0% L1:34.2% BI:30.8%
x264 [info]: 8x8 transform intra:46.2% inter:37.9%
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: coded y,uvDC,uvAC intra: 94.7% 85.5% 76.3% inter: 60.2% 44.5% 21.2%
x264 [info]: i16 v,h,dc,p: 7% 4% 48% 42%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 7% 16% 10% 12% 10% 11% 11% 14%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 7% 11% 11% 13% 11% 12% 12% 18%
x264 [info]: ref P L0: 62.3% 17.9% 10.5% 5.2% 4.1%
x264 [info]: ref B L0: 83.0% 9.8% 4.7% 2.5%
x264 [info]: kb/s:31271.74
encoded 1782 frames, 1.62 fps, 31271.74 kb/sCompletely unable to replicate. In fact, I get completely different results:
$ ./x264 e:\input.avs --threads 12 --preset veryslow --tune grain --crf 18 --level 41 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --subme 9 --bframes 3 -o test.h264 --frames 1782 --fps 24000/1001
avis [info]: 1920x1080 @ 23.98 fps (2860 frames)
x264 [warning]: DPB size (5 frames, 15667200 bytes) > level limit (4 frames, 12582912 bytes)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: frame I:37 Avg QP:16.92 size:148625
x264 [info]: frame P:703 Avg QP:18.91 size: 91952
x264 [info]: frame B:1042 Avg QP:19.11 size: 61184
x264 [info]: consecutive B-frames: 5.3% 34.4% 32.3% 28.0%
x264 [info]: mb I I16..4: 50.0% 43.7% 6.3%
x264 [info]: mb P I16..4: 8.1% 14.2% 2.1% P16..4: 37.3% 7.3% 6.7% 0.1% 0.1% skip:24.2%
x264 [info]: mb B I16..4: 1.4% 4.3% 0.9% B16..8: 33.9% 1.2% 1.1% direct:15.2% skip:42.0% L0:41.2% L1:40.7% BI:18.1%
x264 [info]: 8x8 transform intra:57.9% inter:54.5%
x264 [info]: direct mvs spatial:97.4% temporal:2.6%
x264 [info]: coded y,uvDC,uvAC intra: 90.9% 77.6% 55.5% inter: 32.4% 30.4% 9.4%
x264 [info]: i16 v,h,dc,p: 15% 11% 38% 36%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 11% 12% 10% 12% 10% 12% 11% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 9% 9% 10% 13% 11% 13% 12% 17%
x264 [info]: ref P L0: 59.0% 16.8% 11.8% 6.7% 5.7%
x264 [info]: ref B L0: 78.1% 11.0% 6.8% 4.0%
x264 [info]: kb/s:14411.99

encoded 1782 frames, 0.56 fps, 14411.99 kb/s

Are you sure your build isn't miscompiled?

(Note that this is with my locally changed copy, so there may be slight changes accordingly, but nothing that would account for such an enormous bitrate difference: is it even the same source?)

[ACrowley]

Youve different Results because the Sample was cutted out roughly from the full M2TS. My encode was based on the full M2TS . Chain was M2TS-DGAVC-AVS with Trims-x264.exe
But the Trims in my encode includes the same grainy Scenes as the Sample has, so i thought the Sample will do the same.
The Trims in the AVs are 1782 frames= trim(6957, 8737)
I cant reproduce it on this Sample too!!
But i still get Underflows on the full M2TS with Trims.

So what should i do ? I dont get the underflows everytime at the same Points so its hard to fine the right frames to make a Sample.

I will run the Sample that you have again with the same cmdline with x264 rev1281 Techhouse

EDIT


D:\>x264.exe sample.avs --preset veryslow --tune grain --crf 18 --level 41 --de
block -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --su
bme 9 --bframes 3 -o test.h264 --frames 1782 --fps 24000/1001
avis [info]: 1920x816 @ 23.98 fps (2861 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: frame I:124 Avg QP:17.67 size:115656
x264 [info]: frame P:643 Avg QP:18.51 size: 91338
x264 [info]: frame B:1015 Avg QP:18.79 size: 62508
x264 [info]: consecutive B-frames: 5.3% 26.3% 38.7% 29.7%
x264 [info]: mb I I16..4: 45.9% 49.6% 4.5%
x264 [info]: mb P I16..4: 10.0% 17.8% 3.5% P16..4: 44.6% 11.2% 11.1% 0.2% 0
.1% skip: 1.5%
x264 [info]: mb B I16..4: 2.3% 4.8% 1.8% B16..8: 41.4% 1.9% 1.9% direct:
18.6% skip:27.2% L0:39.7% L1:44.0% BI:16.3%
x264 [info]: 8x8 transform intra:54.1% inter:51.6%
x264 [info]: direct mvs spatial:99.5% temporal:0.5%
x264 [info]: coded y,uvDC,uvAC intra: 94.6% 71.8% 54.7% inter: 44.7% 34.4% 12.5%

x264 [info]: i16 v,h,dc,p: 7% 6% 45% 41%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 9% 9% 11% 13% 11% 13% 12% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 8% 9% 10% 13% 11% 13% 12% 17%
x264 [info]: ref P L0: 56.7% 18.5% 12.4% 6.8% 5.6%
x264 [info]: ref B L0: 77.8% 11.6% 6.7% 4.0%
x264 [info]: kb/s:14694.20

encoded 1782 frames, 2.16 fps, 14694.20 kb/s


###
no underflows on the Sample..damn1 I have to cut out exactly the same Frames for a Sample.
Ah, rev1271 gives VBV underflows too! Its not rev1281 related

[Boolsheet]

I have the same problem, it only underflows over the full trailer. I'm able to reproduce it on a 2 CPU system with 3 threads:

x264.1281.x86.x264nl.exe --bitrate 40000 --vbv-maxrate 40000 --vbv-bufsize 30000 --slow-firstpass -o island.mp4 island.y4m
yuv4mpeg: 1920x1080@24000/1001fps, 0:0
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
mp4 [info]: initial delay 1001 (scale 24000)
x264 [warning]: VBV underflow (frame 1923, -140889 bits)
x264 [warning]: VBV underflow (frame 2278, -166483 bits)
x264 [warning]: VBV underflow (frame 2279, -504531 bits)
x264 [warning]: VBV underflow (frame 2280, -645691 bits)
x264 [warning]: VBV underflow (frame 2281, -273523 bits)
x264 [warning]: VBV underflow (frame 2282, -408667 bits)
x264 [warning]: VBV underflow (frame 2283, -264835 bits)
x264 [warning]: VBV underflow (frame 2284, -138011 bits)
x264 [info]: frame I:164 Avg QP:14.45 size:276560
x264 [info]: frame P:1877 Avg QP:16.69 size:200095
x264 [info]: frame B:378 Avg QP:16.76 size:185598
x264 [info]: consecutive B-frames: 68.2% 27.2% 3.3% 1.2%
x264 [info]: mb I I16..4: 8.5% 70.3% 21.1%
x264 [info]: mb P I16..4: 1.7% 34.0% 5.0% P16..4: 25.7% 19.4% 12.3% 0.0% 0.0% skip: 1.9%
x264 [info]: mb B I16..4: 0.5% 15.0% 2.8% B16..8: 43.3% 4.4% 6.6% direct: 12.6% skip:14.8% L0:38.2% L1:37.1% BI:24.7%
x264 [info]: 8x8 transform intra:81.3% inter:68.5%
x264 [info]: coded y,uvDC,uvAC intra: 96.2% 93.6% 81.0% inter: 68.1% 64.9% 27.2%
x264 [info]: i16 v,h,dc,p: 32% 16% 16% 36%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 11% 12% 34% 6% 7% 6% 8% 6% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 15% 19% 8% 10% 7% 10% 6% 10%
x264 [info]: ref P L0: 53.4% 28.0% 18.6%
x264 [info]: ref B L0: 75.1% 24.9%
x264 [info]: kb/s:38939.71

encoded 2419 frames, 2.42 fps, 38939.71 kb/s

[ACrowley]

Shouldnt it help when we set the VBV Buffer to 40000 insted of 30000? I mean then theres more Room on those frames (=lot of Grain)where the Bitrate exceed very high Peaks(=over 30Mbit), or am i wrong?

EDIT
No, doesnt help
Also i use the fixed rev 1281 techhouse now, nothing misscompiled

[Boolsheet]

More bits to buffer into is always better, however the whole point of VBV is to apply restrictions. A 40 mbit rate and a 30 mbit buffer may be a bit odd, but depending on the algorithm this should be no problem.
And if Dark Shikari doesn't say "use more VBV friendly rates" I'm assuming the algorithms in x264 can (or shoud be able to) handle it. ;)

[ACrowley]

More bits to buffer into is always better, however the whole point of VBV is to apply restrictions. A 40 mbit rate and a 30 mbit buffer may be a bit odd, but depending on the algorithm this should be no problem.
And if Dark Shikari doesn't say "use more VBV friendly rates" I'm assuming the algorithms in x264 can (or shoud be able to) handle it. ;)

30/40Mbit is BluRay Compliant isnt it?


My Sample above was not correct!
Here is the proper Sample from X-Men Origins Wolferine BluRay with exactly those Frames where x264 produces VBV underflows in 1pass CRF and 2pass VBR
http://www.megaupload.com/?d=6IYV2M1V

I run the raw .264 trough dgindex and with a simple avs
AVCSource("F:\Grain Test x264 Underflow\Sample.dga")
Crop(0, 132, -0, -132)

Source: F:\Grain Test x264 Underflow\Sample.avs
Output: F:\Grain Test x264 Underflow\Sample.mkv
Preset: Veryslow
Tuning: Grain
Profile: High
Params: --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

Analayzing source file:
F:\Grain Test x264 Underflow\Sample.avs: 1920x816, 48000/2002 fps, 1767 frames

Resolution: 1920 x 816
Frame No. : 1767
Frame Rate: 24000/1001

x264 0.77.1281M 39f6499
built by techouse on Oct 11 2009, gcc: 4.4.1 (x86_64.core2.Komisar)
Revision: 1281

Commandline for x264:
"C:\AV Tools\x264 Launcher\pipebuf.exe" "C:\AV Tools\x264 Launcher\avs2yuv.exe" "F:\Grain Test x264 Underflow\Sample.avs" -raw - : "C:\AV Tools\x264 Launcher\x264_x64.exe" --preset veryslow --tune grain --crf 18.0 --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3 --output "F:\Grain Test x264 Underflow\Sample.mkv" --frames 1767 --fps 24000/1001 - 1920x816 : 4

x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [warning]: VBV underflow (frame 1330, -699909 bits)
x264 [warning]: VBV underflow (frame 1331, -684299 bits)
x264 [warning]: VBV underflow (frame 1332, -399723 bits)
x264 [warning]: VBV underflow (frame 1333, -678491 bits)
x264 [warning]: VBV underflow (frame 1334, -267403 bits)
x264 [warning]: VBV underflow (frame 1386, -82957 bits)
x264 [warning]: VBV underflow (frame 1387, -493019 bits)
x264 [warning]: VBV underflow (frame 1388, -490147 bits)
x264 [warning]: VBV underflow (frame 1389, -318243 bits)
x264 [warning]: VBV underflow (frame 1390, -451083 bits)
x264 [warning]: VBV underflow (frame 1391, -672099 bits)
x264 [warning]: VBV underflow (frame 1392, -572163 bits)
x264 [warning]: VBV underflow (frame 1393, -555259 bits)
x264 [warning]: VBV underflow (frame 1394, -791331 bits)
x264 [warning]: VBV underflow (frame 1395, -737563 bits)
x264 [warning]: VBV underflow (frame 1396, -964339 bits)
x264 [warning]: VBV underflow (frame 1543, -641449 bits)
x264 [warning]: VBV underflow (frame 1546, -295720 bits)
x264 [warning]: VBV underflow (frame 1547, -491331 bits)
x264 [warning]: VBV underflow (frame 1548, -403827 bits)
x264 [warning]: VBV underflow (frame 1550, -374277 bits)
x264 [warning]: VBV underflow (frame 1551, -154755 bits)
x264 [warning]: VBV underflow (frame 1552, -195995 bits)
x264 [warning]: VBV underflow (frame 1558, -175440 bits)
x264 [warning]: VBV underflow (frame 1560, -84061 bits)
x264 [warning]: VBV underflow (frame 1567, -885299 bits)
x264 [warning]: VBV underflow (frame 1655, -191489 bits)
x264 [warning]: VBV underflow (frame 1656, -1286027 bits)
x264 [warning]: VBV underflow (frame 1657, -715595 bits)
x264 [warning]: VBV underflow (frame 1658, -900603 bits)
x264 [warning]: VBV underflow (frame 1659, -642627 bits)
x264 [warning]: VBV underflow (frame 1660, -834163 bits)
x264 [warning]: VBV underflow (frame 1661, -408835 bits)
x264 [warning]: VBV underflow (frame 1662, -382923 bits)
x264 [warning]: VBV underflow (frame 1664, -396989 bits)
x264 [warning]: VBV underflow (frame 1670, -560776 bits)
x264 [warning]: VBV underflow (frame 1672, -1304621 bits)
x264 [warning]: VBV underflow (frame 1673, -820283 bits)
x264 [warning]: VBV underflow (frame 1674, -996563 bits)
x264 [warning]: VBV underflow (frame 1675, -575971 bits)
x264 [warning]: VBV underflow (frame 1676, -189195 bits)
x264 [warning]: VBV underflow (frame 1677, -726755 bits)
x264 [warning]: VBV underflow (frame 1678, -829131 bits)
x264 [warning]: VBV underflow (frame 1679, -615563 bits)
F:\Grain Test x264 Underflow\Sample.avs: 1920x816, 48000/2002 fps, 1767 frames

x264 [info]: frame I:131 Avg QP:19.98 size:174618
x264 [info]: frame P:586 Avg QP:20.31 size:175412
x264 [info]: frame B:1050 Avg QP:20.76 size:155792
x264 [info]: consecutive B-frames: 5.2% 17.5% 30.6% 46.7%
x264 [info]: mb I I16..4: 41.9% 48.8% 9.2%
x264 [info]: mb P I16..4: 18.0% 26.1% 9.0% P16..4: 26.5% 11.6% 6.7% 0.0% 0.0% skip: 2.2%
x264 [info]: mb B I16..4: 9.2% 10.7% 6.0% B16..8: 32.8% 2.8% 2.4% direct:21.0% skip:15.1% L0:35.0% L1:34.5% BI:30.5%
x264 [info]: 8x8 transform intra:46.1% inter:37.7%
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: coded y,uvDC,uvAC intra: 94.8% 86.6% 75.6% inter: 60.5% 49.0% 20.9%
x264 [info]: i16 v,h,dc,p: 7% 4% 48% 42%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 7% 16% 10% 12% 10% 11% 11% 14%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 7% 11% 11% 13% 11% 12% 12% 18%
x264 [info]: ref P L0: 62.2% 18.0% 10.4% 5.2% 4.2%
x264 [info]: ref B L0: 83.1% 9.7% 4.7% 2.6%
x264 [info]: kb/s:31397.95
encoded 1767 frames, 1.69 fps, 31397.95 kb/s

[shon3i]

30/40Mbit is BluRay Compliant isnt it?Yes it is, but for safetly you maybe use 30/30 to keep CPB Removal buffer @ 0.9.

[ACrowley]

Yes it is, but for safetly you maybe use 30/30 to keep CPB Removal buffer @ 0.9.

I never had Problems with my Sony BDP S350 @ 30/40Mbit

@Darkshikari
Can you test the new Sample above ?
Im sure you must get Underflows in 1 and 2 pass mode

Thx

[shon3i]

Well you now have. I have issues with overflow/underflow when i change CBP removal to 0.4-0.5 with some streams.

[ACrowley]

Well you now have. I have issues with overflow/underflow when i change CBP removal to 0.4-0.5 with some streams.

X men Origins Wolferine is the one and only BluRay Stream until now which produce those vbv underflows.

All other encodes from same builds (and a bit older)with the same Settings are fine without underflows. But i bet there are a lot of grainy Sources which will make the same Problems with x264 and the same Settings now

[shon3i]

I told you i have same problem with maybe one or max two streams IIRC i have same problem with Final Destination 2 BD which is not grainy. Reducing maxrate helped in my case. Anyway i reproduce same behavior with your stream with CRF. So here is trick stream for x264 developers :)

[ACrowley]

I told you i have same problem with maybe one or max two streams IIRC i have same problem with Final Destination 2 BD which is not grainy. Reducing maxrate helped in my case. Anyway i reproduce same behavior with your stream with CRF. So here is trick stream for x264 developers :)

reducing Maxrate ? I thought the underflows are coming from the Restriction . What helps here is encoding without VBV Max.
but i need VBV Max to stay compliant for SA Players

[shon3i]

reducing Maxrate ?
you have underflow, not owerflow.

[ACrowley]

you have underflow, not owerflow.

ah, sorry..but the same underflow with my Sample ..ok

[Dark Shikari]

you have underflow, not owerflow.You have no idea what you're talking about.

Underflow occurs when the encoder runs out of bits--there are no bits left in the buffer, and it exceeds the max rate.

x264 doesn't warn about overflows.

[ACrowley]

i dont want to harry you DarkShikari....but please can you test this new Sample. Im sure you will get VBV underflows
http://www.megaupload.com/?d=6IYV2M1V

[Dark Shikari]

i dont want to harry you DarkShikari....but please can you test this new Sample. Im sure you will get VBV underflows
http://www.megaupload.com/?d=6IYV2M1VYou don't need to post everything twice.

[ACrowley]

You don't need to post everything twice.

Sorry

By the Way
I get a Bitrate with a maximum Peak of 92Mbit and 38Mbit Average on my Sample without VBV Max @CRF 18
92Mbit Peak cant be right/good ?
The highest Peak with older Builds was 49 Mbit on the Casino Royal Bluray

[LoRd_MuldeR]

I get a Bitrate with a maximum Peak of 92Mbit and 38Mbit Average on my Sample without VBV Max @CRF 18
92Mbit Peak cant be right/good ?

If you don't use VBV, there is absolutely no restriction on the peak bitrate. So 92 Mbit/s certainly isn't illegitimate.

Also I see no reason why that peak bitrate shouldn't be right/good, if x264 decided that the source needs it...

[Chengbin]

If you don't use VBV, there is absolutely no restriction on the peak bitrate. So 92 Mbit/s certainly isn't illegitimate.

Also I see no reason why that peak bitrate shouldn't be right/good, if x264 decided that the source needs it...

But the thing is the source (Blu-ray) does that scene under 40mbps at a better quality than x264's xxx mbps peak (natural for reencodes)

[ACrowley]

If you don't use VBV, there is absolutely no restriction on the peak bitrate. So 92 Mbit/s certainly isn't illegitimate.

Also I see no reason why that peak bitrate shouldn't be right/good, if x264 decided that the source needs it...

i know...i thought the Bitrate is a little bit broken out because i never saw such a high Peak in 3 years of x264 encoding...in not one single case. Only now with CRF and latest builds

Also the Peak in 2pass VBR Mode is only 34Mbit "without" VBV limit:) So i dont think it could be normal that CRF 18 reaches 92Mbit, looks like a bit strange/broken to me,particularly 2pass VBR doesnt output VBV underflows here !!??

92Mbps Peak should not running smooth on SA Player,and BluRay H264 Disc doesnt need/use 92 Mbit too, only 40Mbit

Im not a coder, so DarkShikari etc should take a look at it, but for sure theres something wrong here

[LoRd_MuldeR]

i know...i thought the Bitrate is a little bit broken out because i never saw such a high Peak in 3 years of x264 encoding...in not one single case. Only now with CRF and latest builds

Still there's absolutely nothing wrong with that, unless the newer build produces visually worse results at the same size (same average bitrate), compared to the older build.

But I'd assume that the newer (MB-Tree-enabled) build gives better quality :D

Also the Peak in 2pass VBR Mode is only 34Mbit :) So i dont think it could be normal that CRF 18 reaches 92Mbit, looks like a bit strange/broken to me ? ?

Did your 2-Pass encode come out at the same average bitrate as your CRF encode ???

If not, you can't compare them!

92Mbps Peak should not running smooth on SA Player,and BluRay H264 Disc doesnt need/use 92 Mbit too, only 40Mbit

That's why x264 offers VBV for exactly that purpose. If you don't use VBV, this implies that the peak bitrate is unrestricted!

You can't encode with unrestricted peak bitrate and then complain about bitrate peaks ;)

I'm not a coder, so DarkShikari etc should take a look at it, but for sure theres something wrong here

So far I see no indication of something being wrong...

[ACrowley]

yeah, but VBV dosnt work here properly :)

AND:
crf 18 output is bigger then the source:rolleyes:

Source 209 MB
encode crf 18 275 MB

[LoRd_MuldeR]

I get a Bitrate with a maximum Peak of 92Mbit and 38Mbit Average on my Sample without VBV Max @CRF 18
yeah, but VBV dosnt work here properly :)

You contradict yourself. How can VBV not work peroperly, if VBV isn't enabled at all ???

AND:
crf 18 output is bigger then the source:rolleyes:

Source 209 MB
encode crf 18 275 MB

So what? If you only lower the CRF value enough, your re-encoded file will be bigger than any lossy compressed source for sure!

Again: Absoultely nothing unusual so far...

[Dark Shikari]

yeah, but VBV dosnt work here properly :)So far, unable to replicate the problem here on your source...crf 18 output is bigger then the source:rolleyes:Well obviously, if you encode at a quality higher than that of the source, the file will get bigger? :rolleyes:

[shon3i]

So far, unable to replicate the problem here on your source...How you don't? Use exatly same settings like ACrowley. I sucessful every time. Did you use DGAVCdec for indexing .264 file or what? Maybe is there problem?

[Dark Shikari]

How you don't? Use exatly same settings like ACrowley. I sucessful every time. Did you use DGAVCdec for indexing .264 file or what? Maybe is there problem?So far. I'm not done encoding the file; I realized that I hadn't put in the crop command to the Avisynth script the first time around, so I'm running the encode again.

Also, I only have two cores, so using 12 threads might not give entirely the same results.

It takes almost 90 minutes to test, so you can't exactly expect me to come up with this stuff instantly...

[ACrowley]

no i dont contradict myself. I need VBV in x264 to encode x264 for my SA Players. And VBV doesnt work in crf mode without underflows a the the Moment with some/this Source
Thats the topic here, isnt it ? And me an other Users here get VBV underflows...so theres a Problem or not?
Encoding without VBV was only a testrun where i got these Results

The Quants are very high with crf 18
frame I:29 Avg QP:30.17 size:122020
frame P:146 Avg QP:30.21 size: 85443
frame B:262 Avg QP:31.04 size: 65908

So x264 compresses here very bad in CRF Mode but with bigger Filesize/Bitrate as the Source and doesnt look good!
But 2pass VBR ,even with VBV, outputs the same quality with the same quants but at VBR 15mbit and nearly half Filesize?
Mhh.....Ok.

Maybe i should give up....obviously there isnt any Problem for you guys..even other users can reproduce it, we will see

@DarkShikari
The Underflows appear from frame ~1330.

[LoRd_MuldeR]

no i dont contradict myself. I need VBV in x264 to encode x264 for my SA Players. And VBV doesnt work in crf mode without underflows ate the Moment with some Sources.
Thats the topic here, isnt it ? And me an other Users here get VBV underflows...so theres a Problem or not?

Yes, VBV underflows are a problem indeed!

But I was referring to your recent posts where you said that you didn't use VBV and complained about bitrate peaks at the same time :p

You can complain about underflows with VBV enabled, yes. But you can not complain about bitrate peaks without VBV...

[Dark Shikari]

no i dont contradict myself. I need VBV in x264 to encode x264 for my SA Players. And VBV doesnt work in crf mode without underflows ate the Moment with some Sources.
Thats the topic here, isnt it ? And me an other Users here get VBV underflows...so theres a Problem or not?
Encoding without VBV was only a testrun where i got these Results

The Quants are very high with crf 18
frame I:29 Avg QP:30.17 size:122020
frame P:146 Avg QP:30.21 size: 85443
frame B:262 Avg QP:31.04 size: 65908

So x264 compresses here very bad but with bigger Filesize and doesnt look good.Higher quants without VBV than with VBV?

OK, you're just making things up at this point. I give up.

[ACrowley]

Higher quants without VBV than with VBV?

OK, you're just making things up at this point. I give up.

Wait a Minute, maybe i wasnt correct and disturbed a few Things. Quants with CRF 18 ar lower compared with 2pass VBR 15Mbit...ofcourse
I wont talk about "without" VBV here anymore

Sorry, but my English is not very good..i dont want confuse you.
Maybe we should wait if you can reproduce it...

[vucloutr]

well i can reproduce the underflows with ACrowley's sample (http://www.megaupload.com/?d=6IYV2M1V) using x264 x86 r1281 from x264.nl with just these settings:

x264 --crf 15 -o NUL 1080p.avs --vbv-bufsize 30000 --vbv-maxrate 40000 --b-adapt 2 --seek 1300

http://i.imagehost.org/0893/underflow.png (http://i.imagehost.org/view/0893/underflow)

i tried it several times it always starts underflowing from frame 91+ on (which is frame 1391+ without --seek 1300).
same settings with --b-adapt 1 never underflowed.
B-frames average QP is quite interesting.

[Dark Shikari]

Problem finally replicated. Now don't bug me about it for a while and it might eventually get fixed :p

[Dark Shikari]

OK, this is a very very old issue related to B-frames not being tracked by VBV. I'll see what I can do, but I'm not sure there's that much that I can do...

[woah!]

using same settings but with -B 32000 instead of --crf 18 doesnt have a problem.... if you want a copy with such a high bitrate that the result is 1.5x bigger than the source... what SAP are you using?

[Dark Shikari]

Problem fixed by adding row-based ratecontrol to B-frames. Locally committed. Thanks for the test sample.

[Blue_MiSfit]

Awesome :)

[shon3i]

Problem fixed by adding row-based ratecontrol to B-frames. Locally committed. Thanks for the test sample.
Did this have effect on 2pass encoding too?

[Dark Shikari]

Did this have effect on 2pass encoding too?Yes, but probably not as significant as 1-pass.

[ACrowley]

using same settings but with -B 32000 instead of --crf 18 doesnt have a problem.... if you want a copy with such a high bitrate that the result is 1.5x bigger than the source... what SAP are you using?

The Full Source with CRF 18 doesnt reach 92Mbit and 38Mbit average ofcourse...and its smaller then the Source.
The Sample should ony represent the excessive Amount of Film Grain + its VBV Problem so the Bitrate is so high.
2pass VBR ~14800Mbps +DTS 768Kbps Audio reaches 12617Mb(1xDVD9+1XDVD5) in this Case, what is my Target here.

I encode Level 4.1 compliant for my Popcornhour A110. Usually 1080p with Bitrates ~8-15Mbps VBR and mostly the Peak doesnt reach 40Mbit, even without VBV
Or i encode AVCHD /Bluray compliant for my Sony BDP S350

Problem fixed by adding row-based ratecontrol to B-frames. Locally committed. Thanks for the test sample.

Thank you so much..nice!:)

[Dark Shikari]

And both sets of fixes are committed. Enjoy.

[ACrowley]

And both sets of fixes are committed. Enjoy.

So the 1286 is fixed ,thx again..great work!

Ah, i see
http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog

EDIT
Works without VBV Underflows:) (Test Sample from frame 1330)

Source: F:\Grain Test x264 Underflow\Sample.avs
Output: F:\mbn.mkv
Preset: Veryslow
Tuning: Grain
Profile: High
Params: --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

Analayzing source file:
F:\Grain Test x264 Underflow\Sample.avs: 1920x816, 48000/2002 fps, 437 frames

Resolution: 1920 x 816
Frame No. : 437
Frame Rate: 24000/1001

x264 0.77.1286M bcc9a4b
built by techouse on Oct 12 2009, gcc: 4.4.1 (x86_64.core2.Komisar)
Revision: 1286

Commandline for x264:
"C:\AV Tools\x264 Launcher\pipebuf.exe" "C:\AV Tools\x264 Launcher\avs2yuv.exe" "F:\Grain Test x264 Underflow\Sample.avs" -raw - : "C:\AV Tools\x264 Launcher\x264_x64.exe" --preset veryslow --tune grain --crf 18.0 --level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 40000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3 --output F:\mbn.mkv --frames 437 --fps 24000/1001 - 1920x816 : 4

x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
F:\Grain Test x264 Underflow\Sample.avs: 1920x816, 48000/2002 fps, 437 frames

x264 [info]: frame I:9 Avg QP:21.09 size:284658
x264 [info]: frame P:141 Avg QP:21.52 size:243902
x264 [info]: frame B:287 Avg QP:23.27 size:164974
x264 [info]: consecutive B-frames: 1.4% 13.6% 42.1% 43.0%
x264 [info]: mb I I16..4: 52.7% 30.7% 16.6%
x264 [info]: mb P I16..4: 23.5% 25.6% 13.0% P16..4: 18.5% 10.9% 5.3% 0.0% 0.0% skip: 3.2%
x264 [info]: mb B I16..4: 5.4% 13.0% 5.4% B16..8: 33.1% 3.1% 2.5% direct:23.4% skip:14.1% L0:33.4% L1:24.7% BI:41.9%
x264 [info]: 8x8 transform intra:46.1% inter:30.4%
x264 [info]: direct mvs spatial:98.6% temporal:1.4%
x264 [info]: coded y,uvDC,uvAC intra: 95.9% 91.7% 83.3% inter: 68.3% 48.0% 23.9%
x264 [info]: i16 v,h,dc,p: 7% 3% 53% 37%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 8% 17% 10% 11% 10% 11% 12% 15%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 6% 7% 11% 11% 12% 10% 12% 12% 19%
x264 [info]: ref P L0: 69.0% 14.2% 8.3% 4.6% 3.9%
x264 [info]: ref B L0: 87.0% 6.9% 3.5% 2.6%
x264 [info]: kb/s:37000.86
encoded 437 frames, 1.35 fps, 37000.85 kb/s

[laserfan]

And both sets of fixes are committed. Enjoy.Amazing! And to think that some of us spent our Sunday flat-on-our-bunions watching golf on TV!
:thanks:

[shon3i]

@ACrowley, did you SAP aslo play smoothly?

[ACrowley]

@ACrowley, did you SAP aslo play smoothly?

yes, works fine on popcornhour A110 ,even the Peak is very high. But the A110 plays all my old unrestricted encodes too.
Ofcourse its a Problem on BluRay Players with such high Peaks over 80Mbps.
Havent test it but i bet my Sony BDP S350 cant play it. So i have to use 2pass with VBV 40Mbps which works fine always

[Blue_MiSfit]

Well, something nasty happened between 1281 and 1287.

I saw these updates and was very happy, so I re-ran my test with the Prestige 1080p trailer from Apple. I ran it through like this (all logs look fine):


x264 prestige.avs --bitrate 1692 --vbv-maxrate 1692 --
vbv-bufsize 1692 --preset medium --tune film --thread-input --output NUL --pass
1 --stats ".stats"
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 3.0
x264 [info]: frame I:77 Avg QP:12.80 size: 33111
x264 [info]: frame P:1743 Avg QP:14.82 size: 10807
x264 [info]: frame B:1877 Avg QP:14.44 size: 4338
x264 [info]: consecutive B-frames: 21.7% 23.0% 13.8% 41.5%
x264 [info]: mb I I16..4: 45.0% 0.0% 55.0%
x264 [info]: mb P I16..4: 30.4% 0.0% 0.0% P16..4: 34.1% 0.0% 0.0% 0.0% 0
.0% skip:35.4%
x264 [info]: mb B I16..4: 2.9% 0.0% 0.0% B16..8: 24.1% 0.0% 0.0% direct:
18.1% skip:54.9% L0:31.4% L1:43.6% BI:25.0%
x264 [info]: coded y,uvDC,uvAC intra: 58.6% 51.8% 33.7% inter: 21.9% 13.8% 2.0%
x264 [info]: i16 v,h,dc,p: 48% 20% 18% 14%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 16% 16% 7% 7% 8% 6% 8% 6%
x264 [info]: kb/s:1532.07

encoded 3697 frames, 26.82 fps, 1532.07 kb/s

x264 prestige.avs --bitrate 1692 --vbv-maxrate 1692 --
vbv-bufsize 1692 --preset medium --tune film --thread-input --output "Prestige_1
692_test.mp4" --pass 2 --stats ".stats"
avis [info]: 720x480 @ 23.98 fps (3697 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [warning]: target: 1692.00 kbit/s, expected: 1566.40 kbit/s, avg QP: 18.296
2
x264 [info]: profile High, level 3.0
mp4 [info]: initial delay 10427 (scale 250000)
x264 [info]: frame I:77 Avg QP:12.20 size: 37829
x264 [info]: frame P:1743 Avg QP:13.97 size: 13650
x264 [info]: frame B:1877 Avg QP:21.64 size: 1024
x264 [info]: consecutive B-frames: 21.7% 23.0% 13.8% 41.5%
x264 [info]: mb I I16..4: 26.0% 49.0% 25.0%
x264 [info]: mb P I16..4: 4.8% 13.7% 4.2% P16..4: 19.0% 13.6% 8.4% 0.0% 0
.0% skip:36.2%
x264 [info]: mb B I16..4: 0.3% 0.2% 0.0% B16..8: 13.3% 0.5% 0.9% direct:
1.7% skip:83.1% L0:40.5% L1:50.0% BI: 9.5%
x264 [info]: 8x8 transform intra:58.2% inter:55.9%
x264 [info]: coded y,uvDC,uvAC intra: 66.1% 68.8% 59.5% inter: 15.7% 12.6% 5.0%
x264 [info]: i16 v,h,dc,p: 59% 22% 6% 13%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 13% 28% 5% 6% 7% 5% 7% 6%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 18% 22% 5% 7% 7% 5% 6% 5%
x264 [info]: ref P L0: 77.2% 14.5% 8.3%
x264 [info]: ref B L0: 89.0% 11.0%
x264 [info]: kb/s:1485.22

encoded 3697 frames, 25.84 fps, 1485.22 kb/s


No more Underflows! Yay!

Unfortunately.... the output video is totally messed up - there are artifacts all over the place that look like broken motion compensation. Uh oh! Sample:
http://i35.tinypic.com/29egih3.jpg

Also, the encoder failed to hit the target bitrate. I've noticed this happens a lot with VBV CBR, and seems to be worse when increasing quality settings. I.E. I get closer to the target bitrate using faster settings, and lower than the target bitrate using slower settings. This isn't a huge complaint - and doesn't have anything to do with 1287, but it's still strange, and worth mentioning this time :)

Halp!

~MiSfit

[Dark Shikari]

fixed

[Blue_MiSfit]

Trying again with 1288...
~MiSfit

[Blue_MiSfit]

Nope.. not fixed yet :)

~MiSfit

[XhmikosR]

I think DS means it's been fixed in rev1289. (test build (http://www.mediafire.com/?twjzndgm2mz), no pathes used despite the "M")

[shroomM]

Works for me. Awesome, just awesome...

[Blue_MiSfit]

Indeed. I'll shut up now!!

Thanks Dark_Shikari!!

[ACrowley]

Mhh, strange

I used VBV 40/40Mbit on the full M2TS from Xmen Origins Wolferine.
--level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 40000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

But Bitrateviewer and ElecardStreamEye counts Peaks over VBV Max (~60Mbps)? I made one other encode with rev1292 and i get exactly 38-39.9 Mbps Peaks

Ah, the Sample from above reaches Peaks far beyond 40Mbps with VBV and CRF 18 ? 2pass VBV stays under VBV Max

So it looks like as x264 simply add some more Bitrate to avoid VBV underflows ?

[Dark Shikari]

Mhh, strange

I used VBV 40/40Mbit on the full M2TS from Xmen Origins Wolferine.
--level 4.1 --deblock -3:-3 --ref 5 --vbv-bufsize 40000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

But Bitrateviewer and ElecardStreamEye counts Peaks over VBV Max (~60Mbps)?Neither are likely measuring the buffer correctly.

[ACrowley]

Neither are likely measuring the buffer correctly.

that means ? VBV code in 1 pass CRF doesnt work properly here?
VBV Underflows are fixed definitely

[Dark Shikari]

that means ? VBV code in 1 pass CRF doesnt work properly here? No, it works just fine. It means that your buffer analyzers aren't using the same bufsize to measure the maxrate as you told x264 to.

[ACrowley]

No, it works just fine. It means that your buffer analyzers aren't using the same bufsize to measure the maxrate as you told x264 to.

Mhh, When the VBV Buffer use what "he" wants, what is the Sense of it ?

How can i solve it ? When i use a 50Mbps Buffer i get ~20Mbit over VBV Maxrate, with 60Mbps Buffer again.
mhhh...i think i really should use 2pass VBV with this Source.
It works and the Maxrate stays under the VBV Limit.

[Dark Shikari]

Mhh, When the VBV Buffer use what "he" wants, what is the Sense of it ?

How can i solve it ? When i use a 50Mbps Buffer i get ~20Mbit over VBV Maxrate, with 60Mbps Buffer again.
mhhh...i think i really should use 2pass VBV with this Source.
It works and the Maxrate stays under the VBV Limit.If x264 didn't say there was an underflow, there wasn't an underflow.

Also make sure that VBV checking tools use the same initial buffer occupancy...

[shon3i]

@ACrowley post picture graph of elecard buffer analyzer. btw Elecard buld graph based on HRD model, not what is real on strem, so maybe we have HRD bug again.

[Chengbin]

Also, bitrate viewer provides several forms. The default is in seconds. You can change it to frames (and I believe one other mode) and it'll give you a drastically different graph.

[LoRd_MuldeR]

I don't think BitrateViewer implements a "true" VBV model. So the "peak bitrate" displayed in BitrateViewer won't tell you anything about VBV compliance.

Can't even see any options to set up the VBV parameters (maximum rate and buffer size). Without those parameters set accordingly, a VBV verification is not possible :rolleyes:

So use a tool like this for the verification: http://forum.doom9.org/showthread.php?t=144409

[ACrowley]

Ok
Hre are the Infos about the encode from the Sample in CRF 18 with VBV maxrate 40Mbit / Buffer 30Mbit

Elecard Buffer Analyzer
http://i.imagehost.org/t/0794/2_5.jpg (http://i.imagehost.org/view/0794/2_5)


file name : F:\Grain Test x264 Underflow\crf18 VBV.track_1.264
file size : 91 968 123 bytes

video format : AVC/H.264
initial CBP removal relay : 0.60 sec
buffer size : 30 000 000 bits
declared bitrate : 40 000 000 bps
declared frame rate : 23.98 fps
bitrate type : VBR
padding : 0 bits


buffer underflow at 1.226 sec. ( -2147483648 bits)

......and so on. a lot of Buffer under and overflows ??!

Elecard Stream Eye

video stream type : AVC
resolution : 1920x816
profile:level : High:4.1
aspect ratio : 7x3(unspecified)
chroma format : 4:2:0
interlaced : no
frames count : 436
duration : 00:00:18.134
frame size max : 455 570
avg : 210 936
avg/max (I) : 268 078 / 449 577
avg/max (P) : 232 469 / 455 570
avg/max (B) : 195 085 / 430 309
min : 23 691

framerate declared : 23.976
real : 23.987

bitrate declared : 40 000 000

bit allocation max : 69 493 360
avg : 40 477 972
min : 18 147 500

-----------
VBV Checker:
NAL HRD INFO:

cpb_cnt_minus1 = 0
bit_rate_scale = 3
cpb_size_scale = 3

bit_rate_value_minus1[0] = 78124 [40000000 bits/s]
cpb_size_value_minus1[0] = 234374 [30000000 bits]
cbr_flag[0] = 0

initial_cpb_removal_delay_length_minus1 = 23
cpb_removal_delay_length_minus1 = 23
dpb_output_delay_length_minus1 = 23
time_offset_length = 0

---END HRD INFO---

So it loosk like the VBV Values are ok ,right ? But the Peak Bitrate is so high in 1pass CRF ?

[shon3i]

Well buffer graph is totaly mess, and you have underflows, but this underflows are not produced by x264 VBV model, but the nal-hrd patch. Since hrd patch is very risky you need now to play with VBV values. I reccomend you to try with 30000/30000 or 40000/40000 since you not follow Blu-Ray compilancy.

[detmek]

I did some test encodings. x264 report nothing but Avinaptic shows that was some underflow problems. I did NOT use VBV, and I used CRF.
Should I worry or Avinaptic shows some false alarms?

[LoRd_MuldeR]

I did some test encodings. x264 report nothing but Avinaptic shows that was some underflow problems. I did NOT use VBV, and I used CRF.
Should I worry or Avinaptic shows some false alarms?

Since when does Avinaptic offer VBV analysis?

Also you explicitly say that you did not use VBV, but then you complain about buffer underflows? That makes no sense.

If you don't even used a buffering model while encoding, how can the buffer underflow ???

[talen9]

Since when does Avinaptic offer VBV analysis?

It does, since a long time ago... actually profile compliancy checking was one of the main factors behind AVInaptic creation :)

Look for "Profile compliancy" in the first setting tab, and the "Profile" tab for choosing which profile to check against; if there's not already a profile which you need, you can define a new one, providing the buffer size, buffer init and max rate.

You'll have to do a full analysis for the profile compliancy to be checked, though.

[neuron2]

OK, but it's still silly to test for VBV compliance when you haven't encoded with VBV enabled.

[detmek]

[ About file ]

Name: Diablo3-CinematicTrailer_EU_ENtechno1.mp4
Date: 14/10/2009 01:50:33
Size: 83,674,440 bytes (79.798 MB)

[ Generic infos ]

Play duration: 00:02:06 (125.791666 s)
Container type: MP4/MOV
Major brand: JVT AVC version 0
Compatible brands: ISO Base Media
Creation time: 12/10/2009 22:53:59 UTC
Modification time: 12/10/2009 22:53:59 UTC
Number of streams: 1
Type of stream nr. 1: video (avc1) {GPAC ISO Video Handler}
Audio streams: 0

[ Relevant data ]

Resolution: VERY HIGH (1920 x 1080)
Width: multiple of 32 (GOOD)
Height: multiple of 8 (16 would be better)
Average DRF quality: MEDIUM (22.725736)
Standard deviation quality: HIGH (1.102093)
Std. dev. weighted mean: HIGH (0.920306)

[ Video track ]

Codec: avc1
Resolution: 1920 x 1080
Frame aspect ratio: 16:9 = 1.777777
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 16:9 = 1.777777
Framerate: 24 fps
Number of frames: 3019
Bitrate: 5319.986623 kbps

[ About H.264 encoding ]

User data: x264
User data: core 77 r1292M e381f6d
User data: H.264/MPEG-4 AVC codec
User data: Copyleft 2003-2009
User data: http://www.videolan.org/x264.html
User data: cabac=1
User data: ref=16
User data: deblock=1:-2:-2
User data: analyse=0x3:0x113
User data: me=umh
User data: subme=7
User data: psy=1
User data: psy_rd=1.0:0.0
User data: mixed_ref=1
User data: me_range=16
User data: chroma_me=1
User data: trellis=0
User data: 8x8dct=1
User data: cqm=0
User data: deadzone=21,11
User data: chroma_qp_offset=-2
User data: threads=3
User data: nr=0
User data: decimate=1
User data: mbaff=0
User data: constrained_intra=0
User data: bframes=16
User data: b_pyramid=0
User data: b_adapt=1
User data: b_bias=0
User data: direct=3
User data: wpredb=1
User data: keyint=250
User data: keyint_min=25
User data: scenecut=40
User data: rc_lookahead=40
User data: rc=crf
User data: mbtree=1
User data: crf=18.0
User data: qcomp=0.60
User data: qpmin=10
User data: qpmax=51
User data: qpstep=4
User data: ip_ratio=1.40
User data: aq=1:1.00
SPS id: 0
Profile: High@L5.1
Num ref frames: 16
Chroma format idc: YUV 4:2:0
PPS id: 0 (SPS: 0)
Entropy coding type: CABAC
Weighted prediction: No
Weighted bipred idc: B slices - implicit weighted prediction
8x8dct: Yes
Number of frames: 3019
Drop/delay frames: 0
Corrupted frames: 0

P-slices: 1432 ( 47.433 %) ############
B-slices: 1550 ( 51.342 %) #############
I-slices: 37 ( 1.226 %)
SP-slices: 0 ( 0.000 %)
SI-slices: 0 ( 0.000 %)

[ DRF analysis ]

Average DRF: 22.725736
Standard deviation: 1.102093
Max DRF: 23

DRF<15: 0 ( 0.000 %)
DRF=15: 7 ( 0.232 %)
DRF=16: 21 ( 0.696 %)
DRF=17: 16 ( 0.530 %)
DRF=18: 16 ( 0.530 %)
DRF=19: 45 ( 1.491 %)
DRF=20: 68 ( 2.252 %) #
DRF=21: 22 ( 0.729 %)
DRF=22: 21 ( 0.696 %)
DRF=23: 2803 ( 92.845 %) #######################
DRF>23: 0 ( 0.000 %)

P-slices average DRF: 22.962290
P-slices std. deviation: 0.426073
P-slices max DRF: 23

B-slices average DRF: 22.578064
B-slices std. deviation: 1.374433
B-slices max DRF: 23

I-slices average DRF: 19.756756
I-slices std. deviation: 1.024178
I-slices max DRF: 20

[ Profile compliancy ]

Profile to check: MTK PAL 6000
Resolution: 1920 x 1080 > 720 x 576
Framerate: 24 <> 25
Buffer underflow: 00:01:22 (frame 1978)
Buffer underflow: 00:01:23 (frame 1996)
Buffer underflow: 00:01:24 (frame 2021)
Buffer underflow: 00:01:25 (frame 2040)
Buffer underflow: 00:01:26 (frame 2055)
Buffer underflow: 00:01:26 (frame 2070)
Buffer underflow: 00:01:27 (frame 2083)
Buffer underflow: 00:01:27 (frame 2096)
Buffer underflow: 00:01:29 (frame 2143)
Buffer underflow: 00:01:31 (frame 2190)
Buffer underflow: 00:01:32 (frame 2201)
Buffer underflow: 00:01:32 (frame 2211)
Buffer underflow: 00:01:33 (frame 2240)
Buffer underflow: 00:01:35 (frame 2272)
Buffer underflow: 00:01:36 (frame 2296)
Buffer underflow: 00:01:36 (frame 2312)
Buffer underflow: 00:01:37 (frame 2335)
Buffer underflow: 00:01:38 (frame 2355)
Buffer underflow: 00:01:39 (frame 2368)
Buffer underflow: 00:01:39 (frame 2384)
Error: Too many violations

This report was created by AVInaptic (18-11-2007) on 15 ott 2009, h 00:07:18


Here is what I mean.

It does, since a long time ago... actually profile compliancy checking was one of the main factors behind AVInaptic creation :)

Look for "Profile compliancy" in the first setting tab, and the "Profile" tab for choosing which profile to check against; if there's not already a profile which you need, you can define a new one, providing the buffer size, buffer init and max rate.

You'll have to do a full analysis for the profile compliancy to be checked, though.
I just saw this. Thanks. Nothing to worry about.

[talen9]

OK, but it's still silly to test for VBV compliance when you haven't encoded with VBV enabled.

Fully agreed :)
I was only pointing it out to Lord_Mulder, that's all.

detmek ... you actually didn't read what LM and neuron2 said just before you posted again, did you? :p

[LoRd_MuldeR]

It does, since a long time ago... actually profile compliancy checking was one of the main factors behind AVInaptic creation :)

Look for "Profile compliancy" in the first setting tab, and the "Profile" tab for choosing which profile to check against; if there's not already a profile which you need, you can define a new one, providing the buffer size, buffer init and max rate.

You'll have to do a full analysis for the profile compliancy to be checked, though.

Alright. Found it now. However this doesn't seem to be implemented for H.264. What they call "Profile" isn't related to H.264's profiles at all ;)

Nonetheless Avinaptic seems to be happy with my test encode that was encoded with VBV enabled...

[ About file ]

Name: vbv_test2.avi
Date: 13/10/2009 16:10:04
Size: 283,739,174 bytes (270.595 MB)

[ Generic infos ]

Play duration: 00:01:14 (73.582077 s)
Container type: AVI
Number of streams: 1
Type of stream nr. 0: video
Audio streams: 0
JUNK: Avidemux

[ Relevant data ]

Resolution: VERY HIGH (1680 x 720)
Width: multiple of 16 (GOOD)
Height: multiple of 16 (GOOD)
Average DRF quality: HIGH (20.672325)
Standard deviation quality: HIGH (1.570324)
Std. dev. weighted mean: HIGH (1.547739)

[ Video track ]

FourCC: H264/H264
Resolution: 1680 x 720
Frame aspect ratio: 7:3 = 2.333333 (~2.35:1)
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 7:3 = 2.333333 (~2.35:1)
Framerate: 24.014 fps
Number of frames: 1767
Stream size: 283,691,541 bytes
Bitrate: 30843.548004 kbps
Qf: 1.061837
Key frames: 10 (0; 150; 191; 430; 929; ... 1745)
Null frames: 0
Min key int: 41
Max key int: 499
Avg key int: 176.7
Delay: 0 ms

[ About H.264 encoding ]

User data: x264
User data: core 76 r1292M e381f6d
User data: H.264/MPEG-4 AVC codec
User data: Copyleft 2003-2009
User data: http://www.videolan.org/x264.html
User data: cabac=1
User data: ref=8
User data: deblock=1:-1:-1
User data: analyse=0x3:0x133
User data: me=umh
User data: subme=10
User data: psy=1
User data: psy_rd=1.00:0.15
User data: mixed_ref=1
User data: me_range=24
User data: chroma_me=1
User data: trellis=2
User data: 8x8dct=1
User data: cqm=0
User data: deadzone=21,11
User data: chroma_qp_offset=-3
User data: threads=6
User data: nr=0
User data: decimate=1
User data: mbaff=0
User data: bframes=5
User data: b_pyramid=0
User data: b_adapt=2
User data: b_bias=0
User data: direct=3
User data: wpredb=1
User data: keyint=500
User data: keyint_min=25
User data: scenecut=40
User data: rc_lookahead=60
User data: rc=crf
User data: mbtree=1
User data: crf=15.0
User data: qcomp=0.60
User data: qpmin=10
User data: qpmax=51
User data: qpstep=4
User data: vbv_maxrate=40000
User data: vbv_bufsize=30000
User data: ip_ratio=1.40
User data: aq=1:1.00
SPS id: 0
Profile: High@L5
Num ref frames: 8
Aspect ratio: Square pixels
Chroma format idc: YUV 4:2:0
PPS id: 0 (SPS: 0)
Entropy coding type: CABAC
Weighted prediction: No
Weighted bipred idc: B slices - implicit weighted prediction
8x8dct: Yes
Number of frames: 1767
Drop/delay frames: 0
Corrupted frames: 0

P-slices: 610 ( 34.522 %) #########
B-slices: 1147 ( 64.912 %) ################
I-slices: 10 ( 0.566 %)
SP-slices: 0 ( 0.000 %)
SI-slices: 0 ( 0.000 %)

[ DRF analysis ]

Average DRF: 20.672325
Standard deviation: 1.570324
Max DRF: 26

DRF<12: 0 ( 0.000 %)
DRF=12: 1 ( 0.057 %)
DRF=13: 0 ( 0.000 %)
DRF=14: 0 ( 0.000 %)
DRF=15: 0 ( 0.000 %)
DRF=16: 1 ( 0.057 %)
DRF=17: 6 ( 0.340 %)
DRF=18: 1 ( 0.057 %)
DRF=19: 0 ( 0.000 %)
DRF=20: 1394 ( 78.891 %) ####################
DRF=21: 63 ( 3.565 %) #
DRF=22: 68 ( 3.848 %) #
DRF=23: 57 ( 3.226 %) #
DRF=24: 50 ( 2.830 %) #
DRF=25: 106 ( 5.999 %) #
DRF=26: 20 ( 1.132 %)
DRF>26: 0 ( 0.000 %)

P-slices average DRF: 20.580327
P-slices std. deviation: 1.466882
P-slices max DRF: 26

B-slices average DRF: 20.748038
B-slices std. deviation: 1.580775
B-slices max DRF: 26

I-slices average DRF: 17.6
I-slices std. deviation: 2.690724
I-slices max DRF: 22

[ Profile compliancy ]

Profile to check: MTK PAL 6000
Resolution: 1680 x 720 > 720 x 576
Framerate: 24.014 <> 25
Min buffer fill: 52%

This report was created by AVInaptic (18-11-2007) on 15 ott 2009, h 00:19:35

Note: What Avinaptic calls "MTK PAL 6000" profile, I had adjusted to 30000000 Bit, 27000000 Bit and 40000000 Bit/s for Buffer Size, Buffer Init and Max Rate under options.

@detmek: Still you cannot check against any VBV parameters, unless you used the same (or even more restrictive) VBV parameter at encoding time :sly:

[detmek]

Sorry, I should write in first post. I didn't test files for VBV compilancy in the first place. I encoded the file with 16 refs and 16 B frames, 1080p, to see if it is posible to decode it using DXVA (9500 GT) on Windows XP SP2.
I used AVInaptic to check video atributes, frames, average and max bitrate ect. And, I saw underflow warrnings. Thats all.
And, yes. I saw what LM and neuron2 wrote. I just didn't know to set AVInaptic correctlly.

[woah!]

Well buffer graph is totaly mess, and you have underflows, but this underflows are not produced by x264 VBV model, but the nal-hrd patch. Since hrd patch is very risky you need now to play with VBV values. I reccomend you to try with 30000/30000 or 40000/40000 since you not follow Blu-Ray compilancy.

is he even using nal-hrd as i dont see it in his cmd line?

[shon3i]

is he even using nal-hrd as i dont see it in his cmd line?
Hmm, good question but elecard will reject stream if there is no hrd present in the strem, that is for sure.

[ACrowley]

is he even using nal-hrd as i dont see it in his cmd line?

no, i used no NAL HRD!


[ About file ]

Name: crf18 VBV.track_1.mkv
Date: 15/10/2009 10:08:27
Size: 91,976,225 bytes (87.715 MB)

[ Generic infos ]

Play duration: 00:00:18 (18.185 s)
Container type: matroska
Creation time: 15/10/2009 08:08:26 UTC
Number of streams: 1
Type of stream nr. 1: video (V_MPEG4/ISO/AVC)
Audio streams: 0
Muxing Application: libebml v0.7.7 + libmatroska v0.8.1
Writing Application: mkvmerge v2.9.8 ('C'est le bon') built on Aug 13 2009 12:49:06

[ Relevant data ]

Resolution: 1920 x 816
Width: multiple of 32
Height: multiple of 16
Average DRF: 22.832568
Standard deviation: 2.096730
Std. dev. weighted mean: 2.083826

[ Video track ]

Codec ID: V_MPEG4/ISO/AVC
Resolution: 1920 x 816
Frame aspect ratio: 40:17 = 2.352941 (~2.35:1)
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 40:17 = 2.352941 (~2.35:1)
Framerate: 23.976024 fps
Stream size: 91,965,507 bytes
Play duration: 00:00:18 (18.184833 s)
Bitrate: 40458.114099 kbps
Qf: 1.077053

[ About H.264 encoding ]

User data: x264
User data: core 77 r1281M 39f6499
User data: H.264/MPEG-4 AVC codec
User data: Copyleft 2003-2009
User data: http://www.videolan.org/x264.html
User data: cabac=1
User data: ref=5
User data: deblock=1:-3:-3
User data: analyse=0x3:0x113
User data: me=umh
User data: subme=9
User data: psy=1
User data: psy_rd=1.0:0.3
User data: mixed_ref=1
User data: me_range=24
User data: chroma_me=1
User data: trellis=2
User data: 8x8dct=1
User data: cqm=0
User data: deadzone=6,6
User data: chroma_qp_offset=-4
User data: threads=6
User data: nr=0
User data: decimate=0
User data: mbaff=0
User data: constrained_intra=0
User data: bframes=3
User data: b_pyramid=0
User data: b_adapt=2
User data: b_bias=0
User data: direct=3
User data: wpredb=1
User data: keyint=250
User data: keyint_min=25
User data: scenecut=40
User data: rc_lookahead=60
User data: rc=crf
User data: mbtree=1
User data: crf=18.0
User data: qcomp=0.80
User data: qpmin=10
User data: qpmax=51
User data: qpstep=4
User data: vbv_maxrate=40000
User data: vbv_bufsize=40000
User data: ip_ratio=1.10
User data: aq=1:0.50
User data: pulldown=0
User data: nal_hrd=
SPS id: 0
Profile: High@L4.1
Num ref frames: 5
Chroma format idc: YUV 4:2:0
PPS id: 0 (SPS: 0)
Entropy coding type: CABAC
Weighted prediction: No
Weighted bipred idc: B slices - implicit weighted prediction
8x8dct: Yes
Number of frames: 436
Drop/delay frames: 0
Corrupted frames: 0

P-slices: 138 ( 31.651 %) ################
B-slices: 274 ( 62.844 %) ################################
I-slices: 24 ( 5.505 %) ###
SP-slices: 0 ( 0.000 %)
SI-slices: 0 ( 0.000 %)

[ DRF analysis ]

Average DRF: 22.832568
Standard deviation: 2.096730
Max DRF: 28

DRF<20: 0 ( 0.000 %)
DRF=20: 1 ( 0.229 %)
DRF=21: 194 ( 44.495 %) ################################
DRF=22: 48 ( 11.009 %) ########
DRF=23: 44 ( 10.092 %) #######
DRF=24: 41 ( 9.404 %) #######
DRF=25: 37 ( 8.486 %) ######
DRF=26: 34 ( 7.798 %) ######
DRF=27: 36 ( 8.257 %) ######
DRF=28: 1 ( 0.229 %)
DRF>28: 0 ( 0.000 %)

P-slices average DRF: 22.789855
P-slices std. deviation: 2.062265
P-slices max DRF: 27

B-slices average DRF: 22.781021
B-slices std. deviation: 2.134921
B-slices max DRF: 27

I-slices average DRF: 23.666666
I-slices std. deviation: 1.624465
I-slices max DRF: 28

This report was created by AVInaptic (18-11-2007) on 15 ott 2009, h 10:09:31


#############
Elecard Buffer Analyzer doesnt output one single Error on the AVC BluRay Sources, but massive Errors on x264 Sample encodes with our without VBV..i think its not big Problem because Elecard output over/underflows on all x264 encodes ive tested so far ,where other Tools report nothing and all encodes are playing just fine on HW Players. Dont know whats up with Elecard Analyzer, it doesnt like x264

[LoRd_MuldeR]

Elecard Buffer Analyzer doesnt output one single Error on the AVC BluRay Sources, but massive Errors on x264 Sample encodes with our without VBV..i think its not big Problem because Elecard output over/underflows on all x264 encodes ive tested so far ,where other Tools report nothing and all encodes are playing just fine on HW Players. Dont know whats up with Elecard Analyzer, it doesnt like x264

1. If you did not use VBV while encoding, then any VBV analyzer is supposed to report errors! If it doesn't, this would be just by chance.

2. However if you encoded with VBV, then there shouldn't be any errors. But are you 100% sure that you used identical VBV parameters when encoding and when testing?

3. According to your log you used x264 r1281. Update to the latest revision! There were several VBV fixes recently...

[Trahald]

I tried to duplicate but couldnt. can you provide the smallest sample you can that duplicates the issue on the most current x264?

no, i used no NAL HRD!Ahh ok.

[ACrowley]

1. If you did not use VBV while encoding, then any VBV analyzer is supposed to report errors! If it doesn't, this would be just by chance.

2. However if you encoded with VBV, then there shouldn't be any errors. But are you 100% sure that you used identical VBV parameters when encoding and when testing?

3. According to your log you used x264 r1281. Update to the latest revision! There were several VBV fixes recently...


please... im not stupid ? Ofcourse i use identical VBV Values, 40/40Mbit and the updated rev1291 (it plays no Role which build i use for this Problem with the Elecard Buffer Analyzer)

And Elecard outputs/detects Buffer under and overflows in 1 and 2 pass mode encodes, On "my" x264 encodes, encodes from "other people", with all x264 Settings (level, Refframes,bframes etc) and ALL x264 builds..it play a Role
On this sample too ofcourse:
http://www.megaupload.com/?d=6IYV2M1V

Its nothing magical, im sure everbody can reproduce it cause i can on 4 different System and other people can too.
I use Elecard Buffer Analyzer 2.0.14730 from ElecardStreamEyeStudio..i dont know whats going on here. Load Elecard Buffer Analyzer and click view/Streaminfo and take a look:)

Maybe (i bet)its a simple Bug/Restriction in Elecard Buffer Analyzer :
Because the Stream Info shows "for every Stream" (Vbv or no VBV, all x264 builds, all Settings): and the graphs limit maximum is 30mbit everytime

video format : AVC/H.264
buffer size : 30 000 000 bits
declared bitrate : 40 000 000 bps
declared frame rate : 23.98 fps
bitrate type : VBR
padding : 0 bits

Maybe the Buffer Analyzer can only handle the Bluray compliant 30/40Mbit and detects Errors for everything over 30Mbps!

[LoRd_MuldeR]

please... im not stupid ? Ofcourse i use identical VBV Values, 40/40Mbit

Good ;)

And Elecard outputs detects Buffer under and overflows in 1 and 2 pass mode encodes, On my x264 encodes, encodes from other "people". with all x264 Settings (level, Refframes,bframes etc) and ALL Builds..doesnt play a Role
--level 4.1 --deblock -3:-3 --ref 4 --vbv-bufsize 40000 --vbv-maxrate 40000 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

Well, if x264 did not report any underflows while encoding, but Elecard does detect underflows with identical VBV parameters, this means:

Either Elecard is doing something wrong when checking the stream -or- x264 didn't report the underflows it encountered. The latter definitely shouldn't be the case!

Did you try an alternative VBV Checker, such as Neuron2's on that stream? If so, did it agree with Elecard's ???

[ACrowley]

Oh, i think i have it:)
It looks like the Elecard buffer Analyzer cant handle Buffersizes over 30mbit and VBV Maxrates over 40Mbit ...as i thought above!
When i feed the Analyzer with encodes which has a lower VBV Buffer/Maxrate (in this case 24Mbit), there are absolutely no Errors:)
And ofcourse a BluRay AVC Streams with 30/40Mbits works without detected under/overflows too.

This is the one and only Problem here....!

@lordmulder
As i say above...VBVchecker,Avinaptic= no Errors!

So its obvious that theres no Problem with x264 :)

[shon3i]

It looks like the Elecard buffer Analyzer cant handle Buffersizes over 30mbit and VBV Maxrates over 40Mbit ...as i thought above!
When i feed the Analyzer with encodes which has a lower VBV Buffer/Maxrate (in this case 24Mbit), there are absolutely no Errors
And ofcourse a BluRay AVC Streams with 30/40Mbits works without detected under/overflows too.Like i said Elecard follow HRD, if no HRD info you can't expect to get proper info. Anyway you can get Over/Underflows with HRD VBV besides real VBV is not disturbed. Encoders can preform bad calculation and write bad HRD model which can cause bad decoding, besides everything in stream is right.

btw i have some very old version and lastest of Buffer Analyser, and both reject streams with broken HRD or without HRD stored in streams. So you graph is false evidence

[ACrowley]

Like i said Elecard follow HRD, if no HRD info you can't expect to get proper info. Anyway you can get Over/Underflows with HRD VBV besides real VBV is not disturbed. Encoders can preform bad calculation and write bad HRD model which can cause bad decoding, besides everything in stream is right.

btw i have some very old version and lastest of Buffer Analyser, and both reject streams with broken HRD or without HRD stored in streams. So you graph is false evidence

However..its likely a Elecard Problem and nothing "really" wrong with x264, isnt it?!
My Elecard versions from Streameye studio doesnt reject the Stream without NAL HRD...But the standalone Buffer Analyzer Version rejects it with a Error Message
Yep, when i encode the Sample with Mainconcept Reference 1.6.1 fully Bluray compliant ,Elecard doesnt output Errors

[shon3i]

However..its a Elecard ProbelmIs not elecard problem but is not x264 aslo.

And definitely it cant handle Buffersizes over 30MbitHmm no. Can handle max as standard can.

i encode with this settings

--profile high --level 4.1 --crf 15 --thread-input --threads 12 --deblock -3:-3 --keyint 24 --min-keyint 2 --direct auto --slices 4 --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 50000 --vbv-maxrate 50000 --rc-lookahead 60 --merange 24 --me umh --no-dct-decimate --no-fast-pskip --sar 1:1 --aud --nal-hrd --output "100009.264" "100009.avs"

http://img391.imageshack.us/img391/7959/74935124.jpg

btw my version is not much newer than yours. And Streameye dosen't need HRD, just Bufferanalyser.

[ACrowley]

yeah, youre right shonai
The Analyzer doenst detect Under/overflows on Streams with proper NAL HRD, with all Buffer/Maxrate Sizes...
x264 with --nal-hrd works in the Analyzer..

Anyway ..i dont care about Elecard....

[nm]

Anyway ..i dont care about Elecard....
But you need NAL-HRD if you care about strict Blu-ray compliance.

[ACrowley]

But you need NAL-HRD if you care about strict Blu-ray compliance.

i know.....i use nal-hrd when i encode for my Bluray Player and no nal-hrd for my popcorn hour :)

[nm]

What's the point of encoding things twice with different settings when the devices have so similar capabilities?

[Shinigami-Sama]

What's the point of encoding things twice with different settings when the devices have so similar capabilities?

heating

[LoRd_MuldeR]

heating

LOL. Today I had to heat my room with the help of my GTX260 and FurMark, because the heating installation of the house was out of order :D

[Chengbin]

Smart!

But in all seriousness, external hard drives are better "heaters". My external hard drive is actually a very efficient heater. Its a decent 50-60C, and only use like 7 watts.

[Dark Shikari]

Smart!

But in all seriousness, external hard drives are better "heaters". My external hard drive is actually a very efficient heater. Its a decent 50-60C, and only use like 7 watts.That just means it's bad at dissipating heat, which actually means that it's a bad heater.

Basic laws of thermodynamics: the amount of power used by a device is the amount of heat that it puts out.

[ajp_anton]

Smart!

But in all seriousness, external hard drives are better "heaters". My external hard drive is actually a very efficient heater. Its a decent 50-60C, and only use like 7 watts.How is a 7W hard drive "more efficient" at heating than a 200W GPU?

[neuron2]

Discussion of heat and thermodynamics here is OT. Please stay on topic.