B-pyramids breaking DPB limits discussion

[tetsuo55]

This thread is a continuation of the discussion in the Psy RDO thread.

The problem is this:

We have noticed in hardware decoders that the DPB limit is sometimes broken causing encodes to hang the decoder.

According to Dark Shikari and Manao this should not be possible as x264 is supposed to drop a REF frame to prevent this(if i understood correctly)

However according to Sergey A. Sablin using B-pyramids requires an extra frame next to the existing b-frame and ref frames, this can cause some encodes to go out of DPB limit and hang the decoder

As i have seen the decoder hangs in the wild i believe this to be the case too. It doesn't always happen because x264 dynamically decides how many ref and b-frames to use so the "problem" only occurs when there are enough ref and b-frames assigned to break the limit.

[Dark Shikari]

02:40 < Manao> Dark_Shikari: Sergey is perfectly right
02:41 < Manao> i never had considered this
02:41 < Manao> with mmcos, you could keep the DPB size to 3, by discarding I0
sooner
02:42 < Dark_Shikari> Manao: yeah, I realized that a bit ago
02:42 < Dark_Shikari> its more subtle than we initially considered02:08 < pengvado> Dark_Shikari: I don't care what happens in the case of packet
loss. use mmco, don't increase dpb
02:08 < Dark_Shikari> pengvado: see
http://forum.doom9.org/showpost.php?p=1169046&postcount=429
02:08 < Dark_Shikari> 1) all levels of h.264 require a dpb of 4 to be supported
02:08 < Dark_Shikari> 2) we should be using dpb == ref+1, not dpb=ref
02:08 < Dark_Shikari> or for pyramid, at least
02:09 < pengvado> yet dxva (which is limited to dpb=4 at 1080p) support pyramid
with ref=4
02:09 < Dark_Shikari> no, the point is with low numbers of refs
02:09 < Dark_Shikari> lower than 4, that is
02:09 < Dark_Shikari> it will break
02:09 < pengvado> no, the point is that I don't care what's spec compliant, I
care what works
02:10 < Dark_Shikari> why not just write the dpb of 4 ?
02:10 < Dark_Shikari> it costs nothing
02:10 < Dark_Shikari> since no known devices work with 3 and don't work with 4
02:10 < pengvado> sure, I can write the correct dpb, I just won't take it into
account for level limits
02:11 < pengvado> but what if you use 4 refs? then the correct dpb is 5
02:11 < pengvado> and yet real decoders that only support dpb=4 will correctly
decode it
02:13 < Dark_Shikari> why is that?
02:14 < pengvado> how should I know? I don't even have a dxva
02:14 < pengvado> perhaps it's because video cards have plenty of memory for
frame buffers, only the h264-specific part of dpb is limited
02:15 < Dark_Shikari> what about other hardware devices?
02:16 < pengvado> the x264 hardware compliance threads don't usually discuss
those, so I have no source of such information03:00 < pengvado> mmco doesn't fix sergei's issue since you can't put mmco in
disposable frames
03:02 < Manao> pengvado: so there's no way of discarding b1 before encoding b3
? then pyramid requires a DPB of 4 whatever the case ?
03:02 < pengvado> hack: make them non-disposable and mmco to discard the
cuurent b-frame too
03:02 < Manao> oh
03:03 < Manao> urgh

[shon3i]

Conclusion from developers chat:

If you want full compatability (which don't cost quality) don't use x264 encoder :)

@All x264 devs:

Did we safe if turn off b-pyramids? i mean did we skip that violation?

[MatMaul]

Did we safe if turn off b-pyramids? i mean did we skip that violation?
or reduce --ref by one when we use b-pyramid ?

[Manao]

Reducing --ref doesn't help. When using bpyramid, x264 can only produce technically invalid files (but almost always properly decoded). Fixing it will be no sweat though, provided the devs want to.

[bob0r]

Reducing --ref doesn't help. When using bpyramid, x264 can only produce technically invalid files (but almost always properly decoded). Fixing it will be no sweat though, provided the devs want to.

Maybe you could make the patch?
I am sure it will be commited then!

[shon3i]

Maybe is just need to remove patch which is included in rev 721.

[audyovydeo]

When using bpyramid, x264 can only produce technically invalid files

according to what specs ? I'm getting confused.
Also, which hardware devices have trouble playing back such streams ? Nvidias ? ATIs ? Intels ? PSPs ? standalones ?
Of all hardware devices that can play back video, computer video cards !!under Windows!! are the only ones concerned by DXVA. Let's have a thread where everyone enters make/model/specs of troublesome hardware, let's have community review, and let's then write to the hardware manufacturers.

x264 MUST be compliant to H.264 standard, otherwise it ceases to exist. However, it has no obligation to comply to the DXVA specs (which is not a standard, just a MS API).

He who sits on the standard can tell the others to p__s off.

cheers
a/v

[tetsuo55]

according to what specs ? I'm getting confused.
Also, which hardware devices have trouble playing back such streams ? Nvidias ? ATIs ? Intels ? PSPs ? standalones ?
Of all hardware devices that can play back video, computer video cards !!under Windows!! are the only ones concerned by DXVA. Let's have a thread where everyone enters make/model/specs of troublesome hardware, let's have community review, and let's then write to the hardware manufacturers.

x264 MUST be compliant to H.264 standard, otherwise it ceases to exist. However, it has no obligation to comply to the DXVA specs (which is not a standard, just a MS API).

He who sits on the standard can tell the others to p__s off.

cheers
a/v

Although megui has release-candidate profiles for DXVA the files that are broken are within H264 specs for any L3.1/L4.1 able decoder.
Most testing is done on DXVA because we don't have any standalone testers. We know for a fact that b-pyramids is broken on PSP, now i'm not sure if PSP simply does not support pyramids or that this problem is the cause.

The following stream should work on EVERY L4.1 hardware decoder:
Resolution 704x400
Ref frames= 16
B-frames = 16
B-pyramid = yes
sar = 1:1
VBVMaxBitrate = 40000
KeyframeInterval = 24

(As far as i can seen on google/wikipedia all decoders are L4.1 except for handheld devices like PSP/PDA)

[Manao]

According to h264's specifications. It's a minor issue, it happens to work with most decoders, yet files are invalid

As for for so called DXVA specs, those don't exist. Any hardware decoder strives to be compliant with a couple profile & level of the H264 standard. When it fails to do so, it's an unwanted bug. They don't invent there own standard since they don't make the encoders. The notable exception being Apple and Sony, which make both encoders and decoders, and so can afford not to respect any specific levels ( and piss off everybody else, but they don't care...)

Maybe you could make the patchI'd rather not, since it would conflict with my job. However, I'm sure Dark_Shikari / pengvado will fix it. Especially since properly fixing it would gain a free 3% bitrate when using b-pyramid with less than 3 references.

[tetsuo55]

The notable exception being Apple and Sony, which make both encoders and decoders, and so can afford not to respect any specific levels ( and piss off everybody else, but they don't care...)


Which they indeed do which is probably the real reason why the PSP does not support b-pyramids and why Quicktime uses crappy settings


I'd rather not, since it would conflict with my job. However, I'm sure Dark_Shikari / pengvado will fix it. Especially since properly fixing it would gain a free 3% bitrate when using b-pyramid with less than 3 references.

Interesting, does that mean that there is a slight penalty when using 3 ref frames? what happens when we up the ref frames to 4?

[akupenguin]

Especially since properly fixing it would gain a free 3% bitrate when using b-pyramid with less than 3 references.
Which frames to keep in DPB is entirely separate from how many. Fixing the former improves compression ratio, fixing the latter costs a ref and thus can only worsen compression.

[buzzqw]

sorry pals but the PS3 isn't a standalone palyer ? and there are plenty of ps3 around (or at lease i have one too)
so.. if some clips are to test.. simply put the link, users with ps3 could test it

BHH

[Manao]

Which frames to keep in DPB is entirely separate from how many. Fixing the former improves compression ratio, fixing the latter costs a ref and thus can only worsen compression.The problem is that x264, when using stricly less than 4 references, doesn't currently allow to use the proper L0 reference frame for the first b-frame of a pyramidal b-gop. Thus why I said "properly" fixing it. As a user, I would want to be able to use bpyramids efficiently, even with only 1 reference frame.

[akupenguin]

The problem is that x264, when using stricly less than 4 references, doesn't currently allow to use the proper L0 reference frame for the first b-frame of a pyramidal b-gop. Thus why I said "properly" fixing it. As a user, I would want to be able to use bpyramids efficiently, even with only 1 reference frame.
Certainly. But that does nothing to fix levels compliance.

[lexor]

sorry pals but the PS3 isn't a standalone player?

The way most people use it, it's not. When you feed it an mp4, ts or an m2ts file, its full blown level compliance mode doesn't kick in. Instead it uses what essentially is 4.1 with arbitrary relaxations of constraints. (that doesn't quite make it 5.1 or any other fixed level).

Now you can use it to test stand alone compatibility, but you'd have to make an actual blu-ray structure disc (using tsmuxer's blu-ray structure output). And you can't cheat by putting it into a AVCHD root folder (that again will avoid profile check), but you must take what tsmuxer outputs and burn it onto a DVD5 as is (with udf2.5 filesystem). I don't believe short clips will work for this either (not sure they'll be recognized as blu-ray) so you gonna have to encode a decent size clip that is practically a movie.

So to test level4.1 compliance of some stream, you'd have to do a lot more work than an average PS3 user would be willing to, and it certainly doesn't land itself to posting short test files on the message board.

[buzzqw]

thanks for explanation lexor

BHH

[tetsuo55]

so basically we are dealing with a 2 edged sword here?

A fast fix would be to change the b-pyramid code so that all the needed frames are always in the DPB, but that still would not fix the fact that the data as presented in the stream is incorrect(and possibly confuses hardware decoders)

Fixing the h264 incompliance problem would reduce the maximum ref frames by 1 which could lead to huge reductions in compressability (especially for 1080p)


Are competing products able to get both 4 ref frames and b-pyramids into the 1080p L4.1 DPB limit?

[Manao]

Dropping the max number of ref from 4 to 3 doesn't lead to a "huge reduction in compressibility". You will hardly notice it.

Sergey showed that with a DPB of 4 and bpyramid, you can't use 4 references on all your frames (but you can use 4 on most of them). So, ideally, the patch should :
increase the DPB to max(4, num-ref) when bpyramid are enabled
check when computing the ref lists of each frame how many references are effectively available

[tetsuo55]

Dropping the max number of ref from 4 to 3 doesn't lead to a "huge reduction in compressibility". You will hardly notice it.

Sergey showed that with a DPB of 4 and bpyramid, you can't use 4 references on all your frames (but you can use 4 on most of them). So, ideally, the patch should :
increase the DPB to max(4, num-ref) when bpyramid are enabled
check when computing the ref lists of each frame how many references are effectively available


I agree.

The encoder only has to reduce ref frames by 1 for those frames that require it (which we know hardly happens)

[Manao]

With bpyramid enabled, it happens every time the encoder uses at least two bframes in a row. That's not "hardly", that's almost always.

[shon3i]

I must ask again, did we safe if turn off completly b-pyramid, and use 4 refs for 1080p?

[tetsuo55]

With bpyramid enabled, it happens every time the encoder uses at least two bframes in a row. That's not "hardly", that's almost always.

If that was the case shouldn't all those 1080p files with ref4 and b-pyarmids be crashing all the time?


I must ask again, did we safe if turn off completly b-pyramid, and use 4 refs for 1080p?

Yes but the gains of b-pyramide are said the be considerable enough to have it enabled whenever possible

[Sergey A. Sablin]

If that was the case shouldn't all those 1080p files with ref4 and b-pyarmids be crashing all the time?

not necessarily - you can't ask expected behavior (even expected crash) in the case of unexpected problem in the stream, especially from different software/hardware decoders.

and btw, the deal is not with b-pyramid and 4 refs, but with b-pyramid itself - any other settings don't matter at all (however content may matter, when adaptive b-frames are used)

[tetsuo55]

Hmm i don't think i have ever seen a file that does not use b-adapt.

i guess i still don't understand the bug then

[Sergey A. Sablin]

Hmm i don't think i have ever seen a file that does not use b-adapt.

i guess i still don't understand the bug then

all non-reference b-frames which come after reference b-frame (and before next P-frame) in display order have no room in DPB for correct output.

[Dr.D]

So what???

[Sharktooth]

Yes but the gains of b-pyramide are said the be considerable enough to have it enabled whenever possible
nope...

[Dark Shikari]

nope...Why "nope"? How about "yes"?

[Sharktooth]

bpyramid efficiency depends on the number of b-frames, so no.... it's not good to have it enabled whenever possible since actually you have to reduce refs or Bs number to enable it and not break the DPB limits in certain conditions.
so it's preferable to rise the number of b-frames or refs and keep b-pyrdamid disabled.

[Dark Shikari]

bpyramid efficiency depends on the number of b-frames, so no.... it's not good to have it enabled whenever possible since actually you have to reduce refs or Bs number to enable it and not break the DPB limits in certain conditions.
so it's preferable to rise the number of b-frames or refs and keep b-pyrdamid disabled.Since when? B-pyramid generally gives vastly more gain than having a single extra ref available for unreferenced B-frames.

[Sharktooth]

maybe b-pyramid can be more efficient than 1 more ref... but i doubt it can be more efficient than 1 more B...

[Dark Shikari]

maybe b-pyramid can be more efficient than 1 more ref... but i doubt it can be more efficient than 1 more B...What does the number of B-frames have to do with this?

B-pyramid has no effect on the max number of B-frames you can use. The number of B-frames (beyond 1) has no effect on the DPB size at all...

[tetsuo55]

How about some hard facts:

Encode the following files and compare file size

file1 Ref=3, bframes=3, b-pyramid =yes
file2 Ref=4, bframes=3, b-pyramid =no

And as a bonus

file3 Ref=3, bframes=16, b-pyramid =yes
file4 Ref=4. bframes=16, b-pyramid =no

[Sharktooth]

What does the number of B-frames have to do with this?

B-pyramid has no effect on the max number of B-frames you can use. The number of B-frames (beyond 1) has no effect on the DPB size at all...
DXVA compatibility. 1080p 4B works...3B+bpyr works... 4B+bpyr doesnt... that's with x264... other encoders may work.
the hard fact has been already proven about a year ago. it's better 4B and no b-pyramid than 3B and b-pyramid.

[Dark Shikari]

How about some hard facts:

Encode the following files and compare file size

file1 Ref=3, bframes=3, b-pyramid =yes
file2 Ref=4, bframes=3, b-pyramid =no

And as a bonus

file3 Ref=3, bframes=16, b-pyramid =yes
file4 Ref=4. bframes=16, b-pyramid =noWhy are you using ref=3 for pyramid, and ref=4 for no pyramid? Why not use ref=4 for both? The DPB size will be the same.

[Sharktooth]

read my above post... and the first post.

[Dark Shikari]

DXVA compatibility. 1080p 4B works...3B+bpyr works... 4B+bpyr doesnt... that's with x264... other encoders may work.
the hard fact has been already proven about a year ago. it's better 4B and no b-pyramid than 3B and b-pyramid.This thread has nothing to do with DXVA compatibility, in fact, it is antithesis to it, because the topic is about a problem where x264 produces output that isn't spec-compliant, but which DXVA is perfectly fine with. If you want to talk about this, start another thread, otherwise I will report you for derailing this topic. Again, this thread has nothing to do with DXVA; you're misleading people into thinking that it does.

And no, 4 b-frames and no pyramid is probably worse than 3 b-frames and pyramid.

[tetsuo55]

This thread has nothing to do with DXVA compatibility, in fact, it is antithesis to it, because the topic is about a problem where x264 produces output that isn't spec-compliant, but which DXVA is perfectly fine with. If you want to talk about this, start another thread, otherwise I will report you for derailing this topic. Again, this thread has nothing to do with DXVA; you're misleading people into thinking that it does.

And no, 4 b-frames and no pyramid is probably worse than 3 b-frames and pyramid.


Actually breaking level compliancy breaks DXVA plaback too. It's just that most cards dont implement levels correctly that most files accidentaly work.

If have problems with all 4ref+bpyramid files on certain driver versions, and those files work fine in other versions

4ref+nopyramid and 3ref+pyramid always works regardless of driver version

[Dark Shikari]

4ref+nopyramid and 3ref+pyramid always works regardless of driver versionAs I said, this has nothing to do with DXVA! The problem mentioned in this thread is that ref 1, 2 and 3 with pyramid result in out-of-spec streams. Since you said 3ref+pyramid always works, this is clearly not a DXVA issue!

[Sharktooth]

the issues are connected since it's not only DXVA but other hardware devices having problems... first blu-ray players models for example...
so probably the problem is 1 and it is reflecting in various issues.

[Sergey A. Sablin]

As I said, this has nothing to do with DXVA! The problem mentioned in this thread is that ref 1, 2 and 3 with pyramid result in out-of-spec streams. Since you said 3ref+pyramid always works, this is clearly not a DXVA issue!

In fact, if you'll once manages to read this thread you'll find, that the problem mentioned in this thread is that any number of refs with pyramid results in non-compliance stream.

[Manao]

Sergey is right. But, in addition to the stream being uncompliant with bpyramid and any number of refs, when the number of refs is <= 3, x264 doesn't use bpyramid quite efficiently, and lose 3% bitrate.

[Dark Shikari]

In fact, if you'll once manages to read this thread you'll find, that the problem mentioned in this thread is that any number of refs with pyramid results in non-compliance stream.But my point stands--DXVA doesn't care about this lack of compliance, so its not a DXVA issue.

[Sergey A. Sablin]

But my point stands--DXVA doesn't care about this lack of compliance, so its not a DXVA issue.

unless you actually know the code of every DXVA video card (which you definitely not), your point simply can't stand - no one standard compliance device/software is ought to decode non-compliance streams - this behavior is all way beyond the spec. if stream is decoded - this is simply a luck.

[Dark Shikari]

unless you actually know the code of every DXVA video card (which you definitely not), your point simply can't standSure it can--basically every single DXVA device out there was tested with such streams (see the tremendously long thread on this topic), and it worked. It doesn't matter what the code is if it worked.

I personally couldn't care less about hardware support outside of DXVA if it wasn't for my job, and we at Avail Media have tested the current pyramid method on every single set-top-box that we support and it works just fine.

So, if you (plural) want this fixed, I would suggest that you (plural) stop pissing me off by derailing threads before I simply declare the problem a "wontfix." I do care about improving pyramid at <4 refs, but that is a different problem.

(Also, there's a tradeoff here; if we resolve Sergey's problem by reducing ref by one with pyramid, you are hurting quality on devices that do support the current method of doing things in order to add support for pyramid on devices that don't. These devices are hypothetical, since despite the fact that the current method isn't spec-compliant, no known devices have issues with it.)

[Manao]

Come on, properly reducing the number of refs (ie, computing the exact number of refs allowed on each frame, which means some if not most of the frame will have access to all the references anyway) won't increase bitrate by more than 0.2%, so don't try do justify yourself by that. You don't want to do it, that's all (and that's fine by me, if I did care I would do it).

[Dark Shikari]

Come on, properly reducing the number of refs (ie, computing the exact number of refs allowed on each frame, which means some if not most of the frame will have access to all the references anyway) won't increase bitrate by more than 0.2%, so don't try do justify yourself by that. You don't want to do it, that's all (and that's fine by me, if I did care I would do it).Wouldn't Sergey's problem only be resolvable if, with B-pyramid, all frames used one less ref?

[shon3i]

I don't know why but every comercial encoder (Ateme, Mainconcept, Elecard, Sonic, CCEHD, etc.) automaticly when i turn on bpyramid decrase refs by one. So all encoders look to be in 100% bluray specs, so there is must some rule about?

[Manao]

I'm surprised you have access to so many commercial encoders. And talking about the one I know (Ateme), I doubt it indicates you what number of reference is actually used.

Dark Shikari : as I stated on irc, not all frames need having one less ref to be spec compliant. If we take back Sergey's example :time 0 1 2 3 4 5
dec: I0 P4 B2 b1 b3 P8
dis: - - I0 b1 B2 b3
dpb: 1 2 3 3 4 3Then only P8 can't use all the frames in the DPB, since when P8 is coded, b3 is still in the DPB, but b3 isn't a reference. So, with 4 refs and a DPB of 4, b1,B2,b3,P4 will have 4 references, and P8 "only" 3.

[Quark.Fusion]

…
Now to the stream (decoding order, numbers are display order):

I0 P4 B2 b1 b3 P8 B6 b5 b7 ...

time 0 1 2 3 4 5
dec: I0 P4 B2 b1 b3 P8
dis: - - I0 b1 B2 b3
dpb: 1 2 3 3 4 3


now mind time stamp 4 - b3 is decoded but can't be displayed, cause B2 shall be displayed at this moment, so b3 shall be stored somewhere, while B2 is displaying and P8 is decoding. thus dpb has to have 4 frame buffers, ie sps->vui->max_dec_frame_buffering shall be equal to 4, but it is 3 - that means decoder simply has no room to store it.

What about this:
time 0 1 2 3 4 5
dec: I0 P2 P4 b1 b3 P8
dis: - - I0 b1 P2 b3
dpb: 1 2 3 3 4 3
b3 is decoded, but P2 should be displayed and DPB has (I0 P2 P4) — 3 refs and we have no room to store b3
Or I don't get it?

[Dark Shikari]

What about this:
time 0 1 2 3 4 5
dec: I0 P2 P4 b1 b3 P8
dis: - - I0 b1 P2 b3
dpb: 1 2 3 3 4 3
b3 is decoded, but P2 should be displayed and DPB has (I0 P2 P4) — 3 refs and we have no room to store b3
Or I don't get it?
Your chart is wrong.
time 0 1 2 3 4 5
dec: I0 P2 P4 b1 b3 P8
dis: - - I0 P2 b1 b3

[Quark.Fusion]

How we can display P2 before b1???

[Dark Shikari]

How we can display P2 before b1???If you meant for b1 to be before P2 (in your chart, it isn't), the chart is:

time 0 1 2 3 4 5
dec: I0 P2 b1 P4 b3 P8
dis: - I0 b1 P2 b3 P4

[Quark.Fusion]

I meant that b1 reference I0, P2 and P4 — or it can't?
Can't frame have more 1 reference forward?
If can't then original chart must be:
time 0 1 2 3 4 5
dec: I0 P4 b1 B2 b3 P8
dis: - I0 b1 B2 b3 P4
dpb: 1 2 2 3 3 3

[Dark Shikari]

I meant that b1 reference I0, P2 and P4 — or it can't?How can b1 reference P4? b1 is between I0 and P2, display order. P4 is in the next minigop! It hasn't been decoded yet.

[Quark.Fusion]

So problem is that b1 have 2 forward references in original example? (edited previous post)

[akupenguin]

Quark.Fusion: you probably meant
time 0 1 2 3 4 5 6 7
dec: I0 P2 P4 b1 P6 b3 P8 b5
dis: - - I0 b1 P2 b3 P4 b5
dpb: 1 2 3 3 3 3 3 3
dpb: I0 I0 I0 I0 P2 P2 P4 P4
dpb: P2 P2 P2 P4 P4 P6 P6
dpb: P4 P4 P6 P6 P8 P8
which has no buffering problem. (At least, that's the correct sequence if you want b3 to have two L1 P-refs too.)

edit: fixed (P6), and added exact dpb contents. offset dpb entries to clarify that they're the state between frames (frame decoding itself is considered instantaneous in the HRD timing model).

[Quark.Fusion]

Quark.Fusion: you probably meant
time 0 1 2 3 4 5
dec: I0 P2 P4 b1 P8 b3
dis: - - I0 b1 P2 b3
dpb: 1 2 3 3 3 3
which has no buffering problem. (At least, that's the correct sequence if you want b3 to have two L1 P-refs too.)
But where to store P8 if we have I0, P2 and P4 in DPB (that b3 references)? (All frames references maximum allowed frames as extreme case)
Edit: actually here b3 is forced to reference P8, b3 can't refrerence I0, P2 and P4 as it delays b3.


Can we test that method: if frame have 2 forward references then select best one, drop another and/or look for one more ref backward, or from other side: select best forward reference, then look for backwards — will it prevent decoding problem?

[akupenguin]

If you want 2 future refs and 3 total dpb, then you get only 1 past ref. b3 references P2, P4, P6. Besides, x264 doesn't currently support 2 future P-ref.

But that does propose a solution to pyramid. Don't reduce the number of refs for any frame, rather consistently use 2 future refs for every disposable b-frame.
time 0 1 2 3 4 5 6 7
dec: I0 P4 B2 b1 P8 b3 B6 b5
dis: - - I0 b1 B2 b3 P4 b5
dpb: 1 2 3 3 4 4 4 4
dpb: I0 I0 I0 I0 I0 I0 B2 B2
dpb: P4 B2 B2 B2 B2 P4 P4
dpb: P4 P4 P4 P4 B6 B6
dpb: P8 P8 P8 P8

[akupenguin]

so it's 2 future references that is source of problem — I'm right?)
No, the problem is that there are non-referenced frames that aren't displayed immediately after they're decoded. Referenced frames are no problem: they're kept for referencing anyway, and the same copy of the frame also serves for the display reorder buffer. But non-referenced frames which have to be kept for reordering displace some referenced frame... except that they don't, because non-referenced frames don't modify the reference list, so you have to make sure that there was already space in the DPB by displacing a frame before decoding the b-frame.

[Quark.Fusion]

If you want 2 future refs and 3 total dpb, then you get only 1 past ref. b3 references P2, P4, P6. Besides, x264 doesn't currently support 2 future P-ref.

But that does propose a solution to pyramid. Don't reduce the number of refs for any frame, rather consistently use 2 future refs for every disposable b-frame.
time 0 1 2 3 4 5 6 7
dec: I0 P4 B2 b1 P8 b3 B6 b5
dis: - - I0 b1 B2 b3 P4 b5
dpb: 1 2 3 3 4 4 4 4
dpb: I0 I0 I0 I0 I0 I0 B2 B2
dpb: P4 B2 B2 B2 B2 P4 P4
dpb: P4 P4 P4 P4 B6 B6
dpb: P8 P8 P8 P8
b3 can't reference 4 frames as two forward P-references isn't supported :)
If two forward P-references is supported, then you must delay b1 as it can reference P8. And if you delay b1 then you don't have room to store it as I0 should go to display.
If you start to display I0 later then you don't have room to store b3.


So before B-pyramid every non-reference frame was displayed immediately? As reordering buffer aka decoding buffer should reserve all space for references or there will be chance to break decoding. Can you show example where B-frame uses all refs without more that 1 forward-reference and should be delayed? I see that without B-pyramid you can't reference more that 1 future frame so you store it in DPB with back-references and back-refences already were displayed.

Questions:
1) How you decide when to start to show frames on screen? Because if you delay that by 1 frame you must also delay decoding and if you start immidiately then you produce wrong video because you display one frame for two frame intervals.
2) How DPB decide which frame to drop from buffer?


As Sergey says you can't delay decoding by the spec — it's right?
yes, it does - this violates the spec. there is simply no room for non-reference B-frame in DPB, cause all buffers are used for reference frames. To output this non-reference B-frame in correct time one need to delay decoding of next P-frame, cause there is no room to decode it to. This will break DPB output timing - which is violation of spec. (to be simple this will cause displaying problems)
If you delay it in one place then here is chance that you delay it again in other place and then you should delay video and produce wrong video because you display one frame for two frame intervals.

…

I see that non-ref frame can't be stored in DPB as every frame that is stored in DPB reduce avaible reference count and if you don't decrease that value by storing non-ref frame you risk in buffer overflow.

So conclusion: DPB size requirement not depends on ref count, b-pyramid and number of b-frames — it depends on how many frames you should store for reordering process and you should introduce decoding reordering check to be compatible to specs. Avoiding two forward-references will solve problem if there no way to delay non-referenced frame.

[akupenguin]

b3 can't reference 4 frames as two forward P-references isn't supported
And neither is that particular frame ordering supported yet, so my statement is consistent. Nonetheless, supporting both wouldn't require a large code change, if I decide to do that.

So before B-pyramid every non-reference frame was displayed immediately?
Yes

Can you show example where B-frame uses all refs without more that 1 forward-reference and should be delayed?
No, because any set of frames which use only 1 forward-ref can and should be put in the "conventional" B-frame order where all B-frames are displayed immediately.
But if you want an example that no one would ever use because it gratuitously wastes DPB size:
time 0 1 2 3 4 5 6 7
dec: I0 P4 b3 b2 b1 P8 b7 b6
dis: - - - I0 b1 b2 b3 P4
dpb: 1 2 3 4 4 3 3 4
dpb: I0 I0 I0 I0 I0 P4 P4 P4
dpb: P4 P4 P4 P4 P8 P8 P8
dpb: b3 b3 b3 b3 b7 b7
dpb: b2 b2 b6

Avoiding two forward-references will solve problem if there no way to delay non-referenced frame.
Avoiding two forward-reverences is just giving up and not using pyramid.

[Quark.Fusion]

Quark.Fusion: you probably meant
time 0 1 2 3 4 5 6 7
dec: I0 P2 P4 b1 P6 b3 P8 b5
dis: - - I0 b1 P2 b3 P4 b5
dpb: 1 2 3 3 3 3 3 3
dpb: I0 I0 I0 I0 P2 P2 P4 P4
dpb: P2 P2 P2 P4 P4 P6 P6
dpb: P4 P4 P6 P6 P8 P8
which has no buffering problem. (At least, that's the correct sequence if you want b3 to have two L1 P-refs too.)

edit: fixed (P6), and added exact dpb contents
See how you have P6 in stream while displaying P2 — you can't place b3 here, so b3 can't reference I0, P2 and P4. You force b3 to reference to P6, also you force b5 to reference to P8, pushing P2 from DPB.

Avoiding two forward-reverences is just giving up and not using pyramid.
Why? You can still reference B-frame backwards.

But if you want an example that no one would ever use because it gratuitously wastes DPB size:[/code]
You forget to use all refs. Current B-pyramind wastes 1 frame in DPB as you must reduce refs or increase DPB for compatibility.

Ok, new chart:time 0 1 2 3 4 5 6 7
dec: I0 P4 B2 b1 P6 b3 P8 b5
dis: - - I0 b1 B2 b3 P4 b5
dpb: 1 2 3 3 3 3 3 3
dpb: I0 I0 I0 I0 P4 P4 P4 P4
dpb: P4 P4 P4 B2 B2 P6 P6
dpb: B2 B2 P6 P6 P8 P8
When displaying B2 you must decode new reference frame as non-reference neither can't be displayed, nor stored. So you must add display order check to encoding process. But without support of referencing two p-frames b3 can't reference 3 frames as I0 pushed out of DPB.

time 0 1 2 3 4 5 6 7 8 9 10 11
dec: I0 P5 B2 b1 B3 P9 b4 B7 b6 XX b8 YY
dis: - - I0 b1 B2 B3 b4 P5 b6 B7 b8 P9
dpb: 1 2 3 3 4 4 4 4 4 4 4 4
dpb: I0 I0 I0 I0 I0 P9 P9 P9 P9 P9 P9 P9
dpb: P5 P5 P5 P5 P5 P5 P5 P5 P5 P5 YY
dpb: B2 B2 B2 B2 B2 B7 B7 B7 B7 B7
dpb: B3 B3 B3 B3 B3 XX XX XX
Look how we forced to code refs P9, B7, XX and YY because we used two forward-references. Is that ok?
Note that b4 can't reference I0, only P9, which is second forward p-frame.



For b-frame to reference 2 frames forward that frame also must be reference frame, so new clause: non-reference b-frame can't reference B-pyramid frame forward, but B-pyramid frame can.

[akupenguin]

Sorry for feeding the troll, I'll let someone else explain to the clueless.

[tetsuo55]

If you want 2 future refs and 3 total dpb, then you get only 1 past ref. b3 references P2, P4, P6. Besides, x264 doesn't currently support 2 future P-ref.

But that does propose a solution to pyramid. Don't reduce the number of refs for any frame, rather consistently use 2 future refs for every disposable b-frame.
time 0 1 2 3 4 5 6 7
dec: I0 P4 B2 b1 P8 b3 B6 b5
dis: - - I0 b1 B2 b3 P4 b5
dpb: 1 2 3 3 4 4 4 4
dpb: I0 I0 I0 I0 I0 I0 B2 B2
dpb: P4 B2 B2 B2 B2 P4 P4
dpb: P4 P4 P4 P4 B6 B6
dpb: P8 P8 P8 P8

So this fixes the need to reduce ref frames by 1 right?
Does it also fix the 3% bitrate compression loss when 3 or less ref frames are used?
Are the 2 even related at all?

Can someone change "DBP" in topic title to "DPB" as it's breaks search by title for keyword? (and add tag to thread)

I fixed the title, not sure what you mean by tag.

Please stop doubleposting, use the edit button.

[akupenguin]

So this fixes the need to reduce ref frames by 1 right?
Yes
Are the 2 even related at all?
No

[tetsuo55]

Originally Posted by tetsuo55
So this fixes the need to reduce ref frames by 1 right?

Yes


Then could you pretty please with sugar on top create/submit this fix?

[Sharktooth]

he must code it firts. no hurry.

[UsedUser]

Am I correct in my reading, that the problem with making room in DPB for B-pyramid only arises if --ref < 4 ?

[tetsuo55]

Am I correct in my reading, that the problem with making room in DPB for B-pyramid only arises if --ref < 4 ?
For 1920x1080 yes, for other resulutions maybe not.

[Dr.D]

So what???
3 weeks passed...
Asking again. Slow.

S-o, w-h-a-t?

[bond]

striked

[G_M_C]

I've tried to read this topic, but i'm all confused (probably because i'm a noob at the technical side of x264's workings ;) ).

But i undestand that " DPB-limit" has something to do with the frame-buffer and the number of frames it can contain, and that it is related to the number of refs. The reported error is that x264 sometimes exceeds the number of referensce-frames that were set by the --refs option, have i understood this right ?

I use the b-pyramids on my blu-ray compliant encodings. And for 1080p encodings i use b-pyramids and refs @ 4; But after reading this tread, i get the feeling that i run the risk of getting faulty streams, because x264 might encode more than the 4 refs i set (maximum allowed for blu-ray @ 1080p). Or was this problem fixed / worked around in more recent builds ? Or am i more safe by disabling b-pyramids for now ?

Sorry to ask more explination, but
/me is confused :confused:
Or do i worry too much about this ;)

[shon3i]

i get the feeling that i run the risk of getting faulty streamsCorrect.

Or am i more safe by disabling b-pyramids for now ?I asked same question on page two http://forum.doom9.org/showthread.php?p=1169733#post1169733, i get answer from tetsuo55, and nobody of developers are not comment on this, so i assume that risk is smaller without b-pyramids, and for example sharktooth's profiles uses one less ref frames. I think is completly safe to use 3 ref's and no b-pyramid.

[tetsuo55]

the problem is:

With b-pyarmid enabled. Some of the b-frames will not be kept in de DPB. This means that randomly a b-frame could be unfindable resulting a temporary corruption of the stream or a complete crash of the decoder.

Even with 1 less ref frame this problem could occur, because it doesn't really have anything to do with ref-frames but by lowering the ref frames by 1 the chances of it occuring get a lot lower because there is space to keep a b-frame in DPB.

My advice is the same as Sharktooth's DONT use B-pyarmids until this is fixed.

(The fix being all b-frames in the pyramid need to be in the DPB too)

[Ranguvar]

IMO b-pyramid is still fine to use...

I don't think anyone's found any player yet that this breaks. In theory, though, players could b0rk on b-pyramid streams right now while still being standards-compliant. So, if you have such a device that fails with b-pyramid (again, I don't think any have been found) then disable b-pyramid. Otherwise, b-pyramid away.

[tetsuo55]

IMO b-pyramid is still fine to use...

I don't think anyone's found any player yet that this breaks. In theory, though, players could b0rk on b-pyramid streams right now while still being standards-compliant. So, if you have such a device that fails with b-pyramid (again, I don't think any have been found) then disable b-pyramid. Otherwise, b-pyramid away.

-My videocard is a good example (random corruption for a few seconds when a b-frame is not found)
-PSP completely refuses to play the stream(might simply not support it at all)
-PS3 users report problems with it

[G_M_C]

IMO b-pyramid is still fine to use...

I don't think anyone's found any player yet that this breaks. In theory, though, players could b0rk on b-pyramid streams right now while still being standards-compliant. So, if you have such a device that fails with b-pyramid (again, I don't think any have been found) then disable b-pyramid. Otherwise, b-pyramid away.

True, and that's the main reason why i asked; My BD30 hasn' t had any problems so far. But it could explain why fast-forwarding sometines fails on certain places in the stream, while it works fine in the rest of the stream (player jumps back to to standard playback on those places).

[tetsuo55]

True, and that's the main reason why i asked; My BD30 hasn' t had any problems so far. But it could explain why fast-forwarding sometines fails on certain places in the stream, while it works fine in the rest of the stream (player jumps back to to standard playback on those places).

A stream with that behaviour IS broken even if the annoyance is only minor

[Ranguvar]

I have been playing videos on the PS3 since forever with b-pyramid, and it's not that the DPB "bug" (for lack of a better word) makes the PSP incompatible - it's that the PSP doesn't support ANY b-pyramid, plain and simple.

[G_M_C]

True, and that's the main reason why i asked; My BD30 hasn' t had any problems so far. But it could explain why fast-forwarding sometines fails on certain places in the stream, while it works fine in the rest of the stream (player jumps back to to standard playback on those places).

A stream with that behaviour IS broken even if the annoyance is only minor

Hmmm, how do I test those particular streams to see if it is really this "DPB-problem" and not something else (rule out other possibillities) ?

[UsedUser]

I use the b-pyramids on my blu-ray compliant encodings. And for 1080p encodings i use b-pyramids and refs @ 4

Even with 1 less ref frame this problem could occur, because it doesn't really have anything to do with ref-frames but by lowering the ref frames by 1 the chances of it occuring get a lot lower because there is space to keep a b-frame in DPB.

My advice is the same as Sharktooth's DONT use B-pyarmids until this is fixed.

(The fix being all b-frames in the pyramid need to be in the DPB too)
This is why I just asked if the problem @ 1080p is ONLY with less than 4 refs.

According to my reading of the thread, and tetsuo, your reply, the encoding settings G_M_C is using are fine.

Using B-pyramid with 4 refs @ 1080p should not produce the problem --- using B-pyramid with 3 refs @ 1080p potentially produces the problem.

So,

1) B-pyramid is OK to use with 4 refs @ 1080p
2) There's no reason to BOTH lower to 3 refs AND disable B-pyramid
3) If you disable B-pyramid, then there's no reason to lower to 3 refs


Am I wrong?

[tetsuo55]

1) B-pyramid is OK to use with 4 refs @ 1080p
2) There's no reason to BOTH lower to 3 refs AND disable B-pyramid
3) If you disable B-pyramid, then there's no reason to lower to 3 refs


Am I wrong?

1. It could work(do you feel lucky?)
2. Correct
3. Correct

the problem does not always occur, it depends on the source

[UsedUser]

For 1920x1080 yes, for other resulutions maybe not.
If the need without a code fix is to have a DPB of at least 4, then why does the resolution even matter?

If I'm reading correctly, a DPB of 4 allows room for all required references.

[tetsuo55]

If the need without a code fix is to have a DPB of at least 4, then why does the resolution even matter?

If I'm reading correctly, a DPB of 4 allows room for all required references.

There is room but x264 doesnt place them all in there(some b-frames are outside of the DPB and cause the problem)

720p files can allow up to 9 ref frames, thats why resolution matters(same problem with b-pyramids though)

No matter how you twist it around this problem causes x264 to create invalid streams. If your hardware even cares about this is another story alltogether. As you have read some people don't notice a thing, others have completely unplayable streams.

All b-pyramids does is make the compression a little more efficient. I hear it takes a little quality to do so.
The gained compression is minimal at best

I recommend not using it until a fix hits the trunk if your target is hardware.

[Sharktooth]

5%???!?!? where did you read that???
that's not even close to the real gain (that is much much much less). it also depends on the number of b-frames used.

[UsedUser]

There is room but x264 doesnt place them all in there(some b-frames are outside of the DPB and cause the problem)

720p files can allow up to 9 ref frames, thats why resolution matters(same problem with b-pyramids though)
OK, sorry --- not getting it. I understand L4.1 allows 9 refs @ 720p, but that doesn't mean the problem occurs.

It appears to me DPB=4 is an absolute floor, above which the problem is not present.

Can someone explain --- set me straight?

[G_M_C]

Well, quite a discussion about ref=4 being OK or not. I wanted to avoid this whole discussion, by just testing if ref=4 is right or not.

So if someone can point me to a way how to test my stream (that are encoded using 3 b-frames, b-pyramids and ref=4 for 1080p) we can all find out if ref=4 is safe or not ;)

[tetsuo55]

The b-pyramid problem occurs regardless of Ref number.

Lowering the ref-frames by 1 reduces the chances of the stream breaking.

Regardless of the stream working or not its still malformed.



5%???!?!? where did you read that???
that's not even close to the real gain (that is much much much less). it also depends on the number of b-frames used.

Removed the 5% and wrote minimal improvement

Well, quite a discussion about ref=4 being OK or not. I wanted to avoid this whole discussion, by just testing if ref=4 is right or not.

So if someone can point me to a way how to test my stream (that are encoded using 3 b-frames, b-pyramids and ref=4 for 1080p) we can all find out if ref=4 is safe or not ;)


I understand your wish to run tests, however it has already been shown in this thread that the bug has to do with x264 encoder choices.

If a stream does or does not work on a particlar hardware is a pure draw of luck.

[G_M_C]

[...]
I understand your wish to run tests, however it has already been shown in this thread that the bug has to do with x264 encoder choices.

If a stream does or does not work on a particlar hardware is a pure draw of luck.

So ?

That has nothing to do with a way to test the stream after x264 has encoded it has it ? Or is there no way to test this, avinaptic or some other app ?

[tetsuo55]

So ?

That has nothing to do with a way to test the stream after x264 has encoded it has it ? Or is there no way to test this, avinaptic or some other app ?

It means that no matter how much you test your tests will be inconclusive.

[Turtleggjp]

5%???!?!? where did you read that???
that's not even close to the real gain (that is much much much less). it also depends on the number of b-frames used.

When I was doing tests with x264 back in June of this year to determine what settings I wanted to use, I noticed that by not using b-pyramid, the size of my encode went down by about 1.644%. I was using CRF 24.0, so I assumed this meant that using b-pyramid was less efficient. I only tried this with my one testing clip, so I don't know if this was a trend or not. The clip was a show I captured in HD, so it wasn't perfectly clean (like say a Blu Ray might be). The clip was pretty long though, around 20 minutes (typical 30 min show, without commercials), but was pretty low motion generally. I was also using trellis=1 at the time of testing, which I have since switched to using trellis=0, since that comes out quite a bit smaller. Not sure if that makes a difference.

[Manao]

Sorry to shout, but INCREASED SIZE AT SAME CRF DOESN'T MEAN THE OPTION IS LESS EFFICIENT. Said otherwise, CRF XX.X gives you a constant quality only if you use the same settings. And so, CRF XX.X with settings Y gives a different quality than CRF XX.X without settings Y. So, if enabling Y increase bitrate but also increasing quality, you can't say Y is less efficient (especially since in bpyramid case, it's actually more efficiency, always).

[Turtleggjp]

Sorry to shout, but INCREASED SIZE AT SAME CRF DOESN'T MEAN THE OPTION IS LESS EFFICIENT. Said otherwise, CRF XX.X gives you a constant quality only if you use the same settings. And so, CRF XX.X with settings Y gives a different quality than CRF XX.X without settings Y. So, if enabling Y increase bitrate but also increasing quality, you can't say Y is less efficient (especially since in bpyramid case, it's actually more efficiency, always).

I was afraid someone would say that. Good to know though. I'll have to try and study the results and see if I can tell any difference then.

[Quark.Fusion]

Frames used by b-pyramid gets lower quant that plain b-frames — this is cause of size increase, but at same time quality is better also.

[ajp_anton]

Will lowering the refs enough guarantee a stream that doesn't break the limit? I will be doing an SD-resolution encode soon that really needs to work as it will be sold on DVDs to people with all sorts of players.

[UsedUser]

Will lowering the refs enough guarantee a stream that doesn't break the limit? I will be doing an SD-resolution encode soon that really needs to work as it will be sold on DVDs to people with all sorts of players.

No. You're better off disabling B-pyramid.

[ajp_anton]

So how does this work then? If I use refs ~5 below max, how is B-pyramid going to fill up those 5 plus one more?

[Comatose]

Eh, well you still don't know for SURE... but if you drop refs by 5 it might not be worth it anymore.
I could be very wrong (b-pyramid off vs refs -5 compression loss), buuuuut yeah.

[Dark Shikari]

Eh, well you still don't know for SURE... but if you drop refs by 5 it might not be worth it anymore.
I could be very wrong (b-pyramid off vs refs -5 compression loss), buuuuut yeah.The entire point of this thread beyond the ref<4 issue is that at ref==X, the DPB listed is actually only enough for X-1 refs. This isn't a major issue, and the vast majority of decoders probably won't complain, but its still potentially a problem.

[Dr.D]

IMO b-pyramid is still fine to use...

I don't think anyone's found any player yet that this breaks. In theory, though, players could b0rk on b-pyramid streams right now while still being standards-compliant. So, if you have such a device that fails with b-pyramid (again, I don't think any have been found) then disable b-pyramid. Otherwise, b-pyramid away.
Could you (or anybody) please explain me, how some player could be ok on a failed b-pyramid stream?
My dummy understanding - if there is no space to keep a frame, this frame will be missing. So, from where the player take the missing frame, from air?
So, if the problem occurs, ANY player should fails.
What's wrong with this logic?

[Dark Shikari]

Could you (or anybody) please explain me, how some player could be ok on a failed b-pyramid stream?
My dummy understanding - if there is no space to keep a frame, this frame will be missing. So, from where the player take the missing frame, from air?
So, if the problem occurs, ANY player should fails.
What's wrong with this logic?Because most players probably don't read the DPB value in the header except for the purpose of confirming the stream is playable at all; they probably allocate the same amount of DPB memory no matter what, since its simpler to do it that way. As such, an invalid value in the DPB field wouldn't cause a problem for a decoder that did such a thing.

[ajp_anton]

Eh, well you still don't know for SURE... but if you drop refs by 5 it might not be worth it anymore.
I could be very wrong (b-pyramid off vs refs -5 compression loss), buuuuut yeah.I'm guessing that activating B-Pyramid gives a bigger quality boost than going from ref=6 to ref=11, but I haven't made any tests yet... =)

[Ranguvar]

Very likely, unless perhaps if you're encoding anime/cartoons/your desktop.