x264 Known Hardware accelleration problems and solutions

[Sagekilla]

It's the maximum decoded picture buffer that determines the limit.
1920x1080x4 = 8294400
1280x720x9 = 8294400

So 900p would have a limit of
8294400/(1600*900) = 5.76 reference frames

Why does this happen? Because hardware decoders are only going to add as much cache into the chip as necessary, otherwise you'll be wasting transistor / increasing die space. Software decoders can just allocate as much memory as they feel like since you already have hundreds of megs of ram.

Yeah, thing is it would be very easy and cheap to implement an (not even high speed is necessary) external 32 MB of RAM with a few leads to the main chip. With that, you can easily fit up to 15 (you'd need an extra 1.7 MB for the 16th one) reference frames for 1080p content.

[CruNcher]

Uhh jesus i feel really dumb now, i solved the problem of the lowcomplexity-highpro-test.mp4 after looking again at it, i could have sworn i create it as Level 4.1 but actually it is Level 5.1 that obvously coused the Black Screen with PowerDVD and Haalis Splitter, tough Profile Level is completly ignored by PowerDVD and both the Bitstream and MP4 Parser. Most probably the same case is with Mainconcepts Splitter when setting Annex-B output mode. That's no bad thing not really standard compliant i think, but it's imitating other Software Player (Mplayer and VLC) with Hardware Playback so you never gonna have problems, because even if the Bitstream says Level 5.1 or anything else it will send it to NVIDIA's/ATI's Hardware Decoder anyway and try to playback it. With Haalis Splitter it takes the Bitstream Profile Level into account and stops Playback instantly. Haali could maybe make a Annex-B output mode that would help alot of people without the need to change the Profile Level and Desktop Hardware Playback in combination with PowerDVDs Decoder :) http://forum.doom9.org/showthread.php?p=1082322#post1082322
Im looking now into the other MKV stuttering problems :)

Btw this also means that PowerDVD is perfect for debuging such Problems as we try to solve here i found other streams that stay black and now can be sure it's not coused by the Bitstream Info as such but the Stream itself or the container arround it :)

Streams that playback on G92 VP2:

lowcomplexity-highpro-test.mp4= Num ref frames 1 1280x720 23.976 fps
Track1.mp4= Num ref frames 2 1280x720 29.97 fps
CrowdCross.40000.mp4= Num ref frames 4 1920x1080 24.00 fps fps
blade.mp4= Num ref frames 13 320x240 23.976 fps
christina-highmorenew.mp4= Num ref frames 1 1280x720 23.976 fps
TheGreatestGame_HD_AVC.mp4= Num ref frames 3 1920x1080 23.976 fps
sample_h264_100kbit.mp4= Num ref frames 2 192x242 30 fps
sample_h264_1Mbit.mp4= Num ref frames 2 380x280 30 fps
batman-ateme-3500k-hp.mp4= Num ref frames 6 1280x544 29.97 fps
empire.mp4= Num ref frames 6 1280x720 59.94 fps
hp2-ateme-450k-hp.mp4= Num ref frames 9 720x304 25 fps
The-Island_PPC-Trailer-High_Profile-x264-AAC-CoreAVC.mp4= Num ref frames 4 320x128 23.976
Tractor_HD-High_Profile-x264.mp4= Num ref frames 4 1920x1080 25 fps
crf20.mp4=Num ref frames 3 1440x1080 30 fps
HDTV.mp4= Num ref frames 2 1440x1088 25 fps

Streams that don't playback on G92 VP2:

Num ref frames 7 1920x816 23.976 fps
Num ref frames 2 1920x1088 29.97 fps


only very few doesn't playback and their not done by me ;)

[akupenguin]

What does "Annex-B output mode" have to do with anything? It certainly isn't related to Levels.

[CruNcher]

I can't tell you the only thing i can say it works if "Annex B output for H.264/AVC" is active and when it isn't it stays dark if the Bitstream Profile Level isn't 4.1, same darkness as with Haalis Splitter :)

[foxyshadis]

Every frame of a 1920x1080 film actually takes up 3060KB, just for reference. Don't forget chroma planes. :p

[UsedUser]

CruNcher, not sure how your post relates to DXVA playback. Are you saying it's the splitter and not the Cyberlink decoder that is imposing a limit of L4.1 for DXVA? Can someone distill this for me?

[CruNcher]

Exactly UsedUser Cyberlinks own (Demuxer/Splitter) Framework is ignoring the Profile Level either Bitstream (RAW) or when in any other Container (tested at least with it's MP4 Parser) :D, but Haalis splitter somehow gives this information to Cyberlinks Decoder and it seems to accept it and then the result is instant playback stop if Level is 5.1 (standard that X264 saves or saved Guis seem to adapt to this and chose 4.0 here). Im not talking about the Stream itself im talking about the level_idc flag in the Bitstream.

[arfster]

Yeah, seems the files need to specifiy 4.2 or cyberlink chokes. This guy seems to have a solution (to the 20fps bug, not the ref frames issue):

http://www.avsforum.com/avs-vb/showpost.php?p=12680502&postcount=3872

Tried it on one file, worked perfectly.

[valnar]

Yeah, seems the files need to specifiy 4.2 or cyberlink chokes. This guy seems to have a solution (to the 20fps bug, not the ref frames issue):

http://www.avsforum.com/avs-vb/showpost.php?p=12680502&postcount=3872

Tried it on one file, worked perfectly.

Would level 4.1 or below work fine?

[CruNcher]

Soon people will realize that there is a new problem with X264 bitstreams (Choppy playback or frame stuttering) it happens somewhere here -> (PBPBBPBBPBBBPBPBPBPBPB) <- couseing frame stuttering (IPPPPPPPPI) <--- is ok as soon as it hit's PBPBBPBPBPB it get's choppy again , and i have no idea why for me it seems to look normal, but it's coming from the bitstream this sample even crashes Mplayer when ariving this and shows frame destruction in VLC. With the G92 it just beginns to stutter here after that PB sequence it playbacks normaly again till it hits the next.

Num ref frames 8:


file type : ES
video stream type : AVC/H.264
resolution : 1280x720
profile:level : High:5.1
aspect ratio : 16x9(unspecified)
interlaced : no
frames count : 2 210
frame size max : 181 878
avg : 22 215
avg/max (I) : 84 716 / 181 878
avg/max (P) : 30 043 / 46 513
avg/max (B) : 6 790 / 14 697
min : 2 535
file size : 49 095 553
--------------------------------------------

framerate declared : 29.97
--------------------------------------------
real : (var) 29.97

bitrate declared : 0
--------------------------------------------
real max : 8 280 354
real avg : 5 326 270
real min : 3 189 049
--------------------------------------------


ok changeing level with h264info to 4.1 fixed the stuttering and also doesn't crash Mplayer or showing Frame Destruction with VLC anymore (demuxed from a MKV) hmm so PowerDVD doesn't really seem to ignore the Level fully or Nvidias Hardware Decoder

Fixed with H264info Hardware Playback works fine now on Nvidias Decoder Core (BSP/VP2) with PowerDVD:


file type : ES
video stream type : AVC/H.264
resolution : 1280x720
profile:level : High:4.1
aspect ratio : 16x9(unspecified)
interlaced : no
frames count : 2 209
frame size max : 182 096
avg : 22 238
avg/max (I) : 84 924 / 182 096
avg/max (P) : 30 070 / 46 535
avg/max (B) : 6 812 / 14 719
min : 2 557
file size : 49 124 422
--------------------------------------------

framerate declared : 29.97
--------------------------------------------
real : (var) 29.97

bitrate declared : 20 000 256
--------------------------------------------
real max : 8 287 307
real avg : 5 331 785
real min : 3 194 323
--------------------------------------------


So not only the Black Screen problems can happen under some circumstances (Parser->Decoder) also random Frame Stuttering and the allways 20 fps playback problems (with B-frame Sequences) can be caused on Hardware if the level_idc flag is to high.

So 2 Playback Problems that can arise now from a to High level_idc flag on Hardware pinpointed. With the extremest (that has nothing todo with the Level) being non playback @ all, because of a to high num_ref_frames flag for a given resolution :)

With this Sample it looks like this on Nvidias Hardware Decoder

1.0 = nope (no playback)
1.1 = nope (no playback)
1.2 = nope (no playback)
1.3 = nope (no playback)
2.0 = nope (no playback)
2.1 = frame errors (all frames)
2.2 = frame errors (all frames)
3.0 = frame errors (all frames) (Should be the target for interoperability with Sonys PSP (480/576) also the target for HD-DVD (480/576) and DVD AVC (480/576) NTSC/PAL)
3.1 = frame errors (all frames)
3.2 = frame errors (all frames)
4.0 = works (Should be the target for interoperability with HD-DVD and Blu-Ray SAPs PS3 and Xbox360 and Desktop Decoders Nvidia/ATI (720/1080))
4.1 = works
4.2 = works
5.0 = frame stuttering (B-frame sequences)
5.1 = frame stuttering (B-frame sequences)

bitrate declared is allways written as 20 000 256 by H264info so wrong for the other Levels bellow 4.0

I think we can be almost sure that if selfmastered X264 HD-DVDs and Blu-Rays playback fine on Nvidias (BSP/VP2) and ATIs (UVD) new Hardware Decoders they should playback fine also on real HD-DVD and Blu-Ray SAPs :) but this is strange as officialy 1280x720p with a framerate of 29.97 isn't supported by HD-DVD @ all only 59.94 NTSC and 50 PAL are (sure this makes no real sense anyway and for sure such mastered 29.97 fps HD-DVDs should work on most SAPs i hope).

I think this also plays a major role when thinking about Domain Transcoding Level 4.0 stuff into a lower level (in the future)

If someone does a X264 encode with High Profile and a resolution of 720/1080 X264 should at least automaticly decide that a level_idc flag of 4.0 is apropiate (could also been done by the Guis themselves by following the decissions of the user (for example resizeing))

[UsedUser]

So as it stands now:

DXVA requires that the DPB size complies with Profile High@L4.1, meaning num_ref_frames = 4@1080p and 9@720p. Resolutions in between those can have different quantities of reference frames as long as {Height * Width * num_ref_frames <= 8294400} holds true (i.e., 1920x864 can have num_ref_frames = 5, 1280x648 can have num_ref_frames = 10). References and everything included, the actual value for DPB size cannot exceed 12,582,912 for L4.1.

The Cyberlink decoder, when using DXVA, requires the level_idc value to be between 4.0-4.2 in order to achieve smooth playback (sans stuttering or 20fps). If the stream is actually compliant, but the level_idc value is set too high (i.e., L5.1), then the level_idc flag can be changed to L4.1 and the stream will be played back smoothly.

Essentially, people need to encode to the PS3 profile, except with B-pyramids disabled. That would set L4.1 in the stream and comply with the L4.1 restrictions for all streams up to 1080p.

Sound right?

[akupenguin]

ok changeing level with h264info to 4.1 fixed the stuttering and also doesn't crash Mplayer or showing Frame Destruction with VLC anymore

What stream? I won't believe that without evidence, given that I have the libavcodec sourcecode right here and it completely ignores level.

bitrate declared is allways written as 20 000 256
That can only be a bug in H264info. x264 never specifies bitrate. The standard says that if bitrate is not present, it shall be inferred as the maximum allowed by the level. 20mbit/s is appropriate for levels 3.2 and 4.0, no others.
Or did you use the x264 HRD patch? In that case, it might be a bug in the patch.


Num ref frames 8:
1.0 = nope (no playback)
1.1 = nope (no playback)
1.2 = nope (no playback)
1.3 = nope (no playback)
2.0 = nope (no playback)
2.1 = frame errors (all frames)
2.2 = frame errors (all frames)
3.0 = frame errors (all frames) (Should be the target for interoperability with Sonys PSP (alot of other Mobile Devices) also the target for HD-DVD (480p) and DVD AVC PAL/NTSC)
3.1 = frame errors (all frames)
3.2 = frame errors (all frames)
4.0 = works
4.1 = works (Should be the target for interoperability with HD-DVD and Blu-Ray SAPs and Desktop Decoders Nvidia/ATI (720/1080))
4.2 = works
That checks out. 4.0 is the minimum that has a big enough DPB for 1280x720x8refs. 2.0 doesn't even have enough DPB for one frame.
5.0 = frame stuttering (B-frame sequences)
5.1 = frame stuttering (B-frame sequences)
That's just weird. Good luck figuring it out ;)

[CruNcher]

What stream? I won't believe that without evidence, given that I have the libavcodec sourcecode right here and it completely ignores level.


Yes, Sorry the Stream is fine with Mplayer and the crash is something really really strange (but really if you have a PC you get used to this strangeness over the years), if the file is named mplayercrash.h264 it works if i rename it to less characters like mplayercras.h264 it crashes after some seconds playback (useing Mplayer for Windows and Smplayer) (jesus).
This must be the funniest bug i ever found :D

ID_VIDEO_CODEC=ffh264
Audio: no sound
Starting playback...
VDec: vo config request - 1280 x 720 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is undefined - no prescaling applied.
[swscaler @ 00DA5310]No accelerated colorspace conversion found
[swscaler @ 00DA5310]SwScaler: using unscaled yuv420p -> rgb24 special converter
VO: [directx] 1280x720 => 1280x720 Planar YV12 [zoom]


MPlayer interrupted by signal 11 in module: decode_video
ID_SIGNAL=11
- MPlayer crashed by bad usage of CPU/FPU/RAM.
Recompile MPlayer with --enable-debug and make a 'gdb' backtrace and
disassembly. Details in DOCS/HTML/en/bugreports_what.html#bugreports_crash.
- MPlayer crashed. This shouldn't happen.
It can be a bug in the MPlayer code _or_ in your drivers _or_ in your
gcc version. If you think it's MPlayer's fault, please read
DOCS/HTML/en/bugreports.html and follow the instructions there. We can't and
won't help unless you provide this information when reporting a possible bug.


That can only be a bug in H264info. x264 never specifies bitrate. The standard says that if bitrate is not present, it shall be inferred as the maximum allowed by the level. 20mbit/s is appropriate for levels 3.2 and 4.0, no others.
Or did you use the x264 HRD patch? In that case, it might be a bug in the patch.


Yes it allways sets 20mbit and yes that's not a x264 bug but H264info and i see the bitrate declared stuff is not really needed to keep Hardware compatibility so it's also not interesting @ all.


BTW im asking myself another question (and ofcourse try to find an answer to it) that is if we have a bitstream allready encoded with a to high num_ref_frame flag what would happen if we change it manualy (Hex Edit) to a lower value ?, for sure there would be no Black screen anymore but what else ? (see top level results). In many situation i think it could work even without showing garbage on some Devices at least that's what i hope.

0000000167640033AC56C05005BB0110000003001000000303C8F18313800000 <--- num_ref_frame = 2
0000000167640033AC564014016EC044000003000400000300F23C60C4480000 <--- num_ref_frame = 3

C05005BB0110000003001000000303C8F18313800000 <--- num_ref_frame = 2
4014016EC044000003000400000300F23C60C4480000 <--- num_ref_frame = 3

A: C0 40
B: 50 14
C: 05 01
D: BB 6E
E: 01 C0
F: 10 44
14: 10 04
18: 03 00
19: C8 F2
1A: F1 3C
1B: 83 60
1C: 13 C4
1D: 80 48
BA: 31 32

but where is the flag stored i can't see 20 = 14 or 30 = 1E anywhere hmmm ?

i think its ABCDE
num_ref_frame 2 = C05005BB01
num_ref_frame 3 = 4014016EC0
num_ref_frame 8 = E2405005BA
num_ref_frame 11 = E3005005BA


it worked :) changed the 11 ref frame to 8 and now it plays on the G92 (some block errors appearing here and there now and then but overall it's watchable :)) don't forget this was thought as a workaround to get most of the old non hardware compatible X264 streams working (it won't play 100% correct and question remains if DSPs accept this), but the result is much better then the level 2.1 results from above ;)
I might should add that this way also Software Player show the garbaged blocks, so you should really only use it if it's absolutely necessary (recoding sometimes can be a better solution) as it's a evil bitstream hack ;)

Btw im fixing not my own streams here i never saw a reason to use so much Ref frames or B-frames ever :D it's plain dumb for real footage.
All the streams i ever made so far with X264 are Hardware Compatible from the core (except Level flag), it's the user behind the tool that makes such errors, and alot of people here warned about it before and now those that didn't listen are getting payed for not hearing to our calls (in the end it hurts everyone that want's to play those streams on SAP's or PS3/Xbox360 without recoding them) ;)
It's sad to see that People didn't thought about future proveness of their streams in combination with Hardware Players and used such insane Encoding settings and still do, that really aren't worth it from a Visual Quality Standpoint, we all have warned for this to happen.

And personaly i think it's the best Time to wake up now and force the users to addhere to save settings either doing it via a even smarter acting X264 or via all the GUIs out there, something has to happen keeping interoperable with Hardware Devices or it will continue this way and that can't be the Goal.
Aku i know how you think about forceing, but i think it' absolutely necessary especialy for users that dunno what they do at least the GUI Developers should take this advice finaly into consideration, because you guys are the interface to most users that could create such problematic streams.

[tetsuo55]

What stream? I won't believe that without evidence, given that I have the libavcodec sourcecode right here and it completely ignores level.


That can only be a bug in H264info. x264 never specifies bitrate. The standard says that if bitrate is not present, it shall be inferred as the maximum allowed by the level. 20mbit/s is appropriate for levels 3.2 and 4.0, no others.
Or did you use the x264 HRD patch? In that case, it might be a bug in the patch.


I think he means level instead of bitrate, i can also confirm that changing the level of a non working file will almost always fix DXVA or PS3 playback.

MKV's floating around on the net can be made compatible by changing the level and then rebuilding to mkv or mp4, most of these where encoded with all kinds of incompatible settings i believe.

there will need to be more tests, but it seems that the only thing causing playback failures is the level in the header, not the level used to encode.

So if someone want to try, encode a file with all the settings you would normally use to create a compatible file, but change the level from 4.1 to 5.1.
When the file is finished, use thishttp://www.avsforum.com/avs-vb/showpost.php?p=12680502&postcount=3872 guide to change the header from 5.1 to 4.1 and then test the file in powerdvd

If that actually works, the same can be tried with ref frames above the limit (DXVA should be using the videocard ram, so that means most people will have 256MB at their disposal)

If that works too, the maximum limit for standalone players, ps3 and dxva has to be found, but i don't think we will get this far

-------------

PS. Seeing as CruNcher foudn that SD h264 also does not get DXVA we should add that to the list of things to fix, maybe setting the level to 4.1 (during encode or afterwards) will fix that too?

[CruNcher]

I think he means level instead of bitrate, i can also confirm that changing the level of a non working file will almost always fix DXVA or PS3 playback.

MKV's floating around on the net can be made compatible by changing the level and then rebuilding to mkv or mp4, most of these where encoded with all kinds of incompatible settings i believe.

there will need to be more tests, but it seems that the only thing causing playback failures is the level in the header, not the level used to encode.

So if someone want to try, encode a file with all the settings you would normally use to create a compatible file, but change the level from 4.1 to 5.1.
When the file is finished, use thishttp://www.avsforum.com/avs-vb/showpost.php?p=12680502&postcount=3872 guide to change the header from 5.1 to 4.1 and then test the file in powerdvd

If that actually works, the same can be tried with ref frames above the limit (DXVA should be using the videocard ram, so that means most people will have 256MB at their disposal)

If that works too, the maximum limit for standalone players, ps3 and dxva has to be found, but i don't think we will get this far

-------------



I can't agree with this i definately have streams here created with X264 that do not playback on Hardware (in this case Nvidias 2nd Generation Decoder Core BSP/VP2), because they used to insane encoding settings the badest is a 1280x720p encode that uses 11 ref frames and is real footage.

And no SD H264 shouldn't be set to level 4.1 that's also insane 3.x seems to be quiet ok for such purposes.


PS. Seeing as CruNcher foudn that SD h264 also does not get DXVA we should add that to the list of things to fix, maybe setting the level to 4.1 (during encode or afterwards) will fix that too?


If you looked in my test of streams you gonna see that all the SD and bellow stuff gets fully accellerated without problems
http://forum.doom9.org/showthread.php?p=1082369#post1082369

[akupenguin]

Aku i know how you think about forceing, but i think it' absolutely necessary especialy for users that dunno what they do at least the GUI Developers should take this advice finaly into consideration, because you guys are the interface to most users that could create such problematic streams.
http://mailman.videolan.org/pipermail/x264-devel/2008-January/003922.html
discuss.

[tetsuo55]

I can't agree with this i definately have streams here created with X264 that do not playback on Hardware (in this case Nvidias 2nd Generation Decoder Core BSP/VP2), because they used to insane encoding settings the badest is a 1280x720p encode that uses 11 ref frames and is real footage.

And no SD H264 shouldn't be set to level 4.1 that's also insane 3.x seems to be quiet ok for such purposes.



If you looked in my test of streams you gonna see that all the SD and bellow stuff gets fully accellerated without problems
http://forum.doom9.org/showthread.php?p=1082369#post1082369

Ah i read your post too fast, i now see that correctly encoded SD streams get accelerated.
I wonder what settings are used to encode scene releases then, seeing as fixing the levels seems to fix all their issues.

You are probably right that they (accidentaly?) never go over the ref frames limit for the used resolutions

http://mailman.videolan.org/pipermail/x264-devel/2008-January/003922.html
discuss.

What that guy is saying, is a better explained version of what i mean, however instead of randomly changing x264 i suggest that we find the limits first.

[akupenguin]

What that guy is saying, is a better explained version of what i mean, however instead of randomly changing x264 i suggest that we find the limits first.
What limits? The proposal doesn't depend on any particular player, only on the observation that underestimating playback requirements has less severe repercussions than overestimating.

[tetsuo55]

i did someresearch and i thik this is the correct conclusion:
these are the settings for max quaility + Hardware support.


SD up to DVD(480p ntsc, 576p PAL): High= Profile + Level 3

between 481p ntsc, 577p PAL and 720p : High10 profile + Level 3.1

721p to 1088p: High10 profile + level 4.1

In all cases the ref limit must be taken into consideration

It would seem that high10 is backwards compatible with high, so it should work on all players, also it will only use the 10bit plane if it's actually in the source, or so it seems.

x264 does not encode to high10(yet?) unless it was recently added, so in the case of x264 select high instead is the best quality we're going to get.

I can see why players refuse to play 5.1 files, that level is meant for "2304p"


------

What i can't find out though is, why would it be wrong to transcode a DVD with "high(10) + Levels 4.1 as any HD player should support it.
The only thing i can think of is maybe that the endfile will be bigger?

[akupenguin]

And no SD H264 shouldn't be set to level 4.1 that's also insane 3.x seems to be quiet ok for such purposes.
Why is it insane for an SD file to exceed 20mbit/s peak bitrate? DVD peaks at 10mbit/s, and that's not enough. Even if H.264 compresses better by a factor of 2 (which is an average, not necessarily applicable to the hardest scenes), I won't claim that 20mbit/s is always transparent.

I can see why players refuse to play 5.1 files, that level is meant for "2304p"
It's reasonable for players to reject 2304p files. That doesn't explain why they reject 480p level 5.1 files. All of the information related to whether a given file is playable (such as DPB) is present in the header; level is not the only bit of information a player has to decide based on. A level is label for a collection of features (each of which is also individually specified), conveniently tabled in the standard to help decoder implementers decide on a common set of features to support, and to provide a shorthand for decoders to say what they support. It was never intended to be used as part of the decoding process.

What i can't find out though is, why would it be wrong to transcode a DVD with "high(10) + Levels 4.1 as any HD player should support it.
What makes you think it is wrong? None of the other threads mentioned any minimum resolution.

[tetsuo55]

Well in that case its the players/decoders that are "Wrong" by only checking the "level" to decide wether or not the file will work.

Unless this is enforced in the h264 specifications.

Also it would seem then that everything could be encoded with high10 + 5.1 + ref frames limit for used resolution, the outputted file just has to keep saying its 4.2 (the highest compatible level, so the player can (if it even does at all) set up itself for the highest level it can take)

[UsedUser]

there will need to be more tests, but it seems that the only thing causing playback failures is the level in the header, not the level used to encode.
That's not true. I've demonstrated with my tests that even if a stream declares L4.1 in level_idc, the DPB size (num_ref_frames*resolution) is important for DXVA. The limits for DPB size still exist.

As I just summed up yesterday (http://forum.doom9.org/showthread.php?p=1083095#post1083095), DXVA requires DPB size comply with High@L4.1. Smooth playback sans stuttering/20fps requires the stream's level_idc declare L4.1. Both need to comply for smooth DXVA playback.

[akupenguin]

Well in that case its the players/decoders that are "Wrong" by only checking the "level" to decide wether or not the file will work.
Unless this is enforced in the h264 specifications.
A bitstream specification says what a stream means, i.e. what decoded pixel values correspond to any given compressed stream. It does not and cannot specify any higher-level behavior of a player, i.e. what a player should do with a file.
So, the players aren't violating the standard by rejecting certain streams, because the standard doesn't say anything about that. They are just following suboptimal behavior in terms of what the user probably wants ("play my movie").

[tetsuo55]

That's not true. I've demonstrated with my tests that even if a stream declares L4.1 in level_idc, the DPB size (num_ref_frames*resolution) is important for DXVA. The limits for DPB size still exist.

As I just summed up yesterday (http://forum.doom9.org/showthread.php?p=1083095#post1083095), DXVA requires DPB size comply with High@L4.1. Smooth playback sans stuttering/20fps requires the stream's level_idc declare L4.1. Both need to comply for smooth DXVA playback.

So do you think Encoding a file with High@5.1 while keeping the DPB size limit of high@4.1 and then afterwards changing the the level_idc to 4.1 will work? And even if it does, does it matter for the quality of the encode?

A bitstream specification says what a stream means, i.e. what decoded pixel values correspond to any given compressed stream. It does not and cannot specify any higher-level behavior of a player, i.e. what a player should do with a file.
So, the players aren't violating the standard by rejecting certain streams, because the standard doesn't say anything about that. They are just following suboptimal behavior in terms of what the user probably wants ("play my movie").

Okay in that case, i agree with the mailinglist poster that x264(or frontend) should calculate the lowest level_idc that the stream complies to and apply that to the file.

Besides that we should be a preset for hardware compatibility

[akupenguin]

So do you think Encoding a file with High@5.1 while keeping the DPB size limit of high@4.1 and then afterwards changing the the level_idc to 4.1 will work? And even if it does, does it matter for the quality of the encode?
Yes it will work, and the resulting file will be bitwise identical to encoding at level 4.1. Because x264 doesn't do anything with level_idc other than write it to the file.

[CruNcher]

Yep but sooner or later X264 has to take full VBV and HRD into account when the next problems should appear then with Hardware Decoders, if that ever happens ofcourse, but the problems arised with XviD @ that time are a good indicator that this could happen anytime again.
Surely the Lowcost DSPs in the begining of that time aren't really comparable with AVC DSPs nowdays, wich are much cleaner designed and support everything from the start (and for sure have more resources left to compensate problematic situations for a longer time then those DSPs back then could).
Also H264s bitrate behaviour even without any VBV seems really strict allready so i doub't this will make any problems @ all, at least with lower bitrate encoding stuff (Professional Mastering of HD-DVD or Blu-Rays could give quiet a different picture, but that's anyways not what X264 was made for and other stuff is available from DivX/Mainconcept (Film) or Ateme (Broadcast) for that purpose wich are a step more Strict in that direction, those btw allready write since a long time the correct level_idc into their bitstreams taking resolution and dpb into account).
But yes i would suggest that when DPB size is in the known parameters, especialy for HD that X264 then writes the given level_idc flag for this situation, but also warns the user if it exceeds those "with a clear message" that this most probably won't playback on Hardware Devices. The GUIs have todo the same warning, not everyone will ever see the X264 cmd where this is shown, this way nothing is forced yet and the user are knowing about the problems that they could experience and can decide themselves if they wan't to riks them or not. Personaly i think that's the nicest solution for now for everyone (especialy the Anime Guys will love that, those for sure sometime in the not so distant future are gonna crowed Doom9 with questions like: "why is my Stream not playing on my HD-DVD/PS3/Xbox360/PSP/ipod" i fear) ;)

[tetsuo55]

These would seem to be the maximum quality settings, (something could go wrong with the b-pyramid one's as i have not tested to see if their end result is more than 2 (this could be the case in the SD files))

I believe these are the best settings for hardware accelerated h264 files, please correct me if i made a mistake

In ALL cases, profile must be High@4.1

1920x1080p --ref 1 --bframes 3 --b-adapt --b-pyramid
1920x1080p --ref 3 --bframes 3 --b-adapt --no-b-pyramid

1280x720p --ref 6 --bframes 3 --b-adapt --b-pyramid
1280x720p --ref 8 --bframes 3 --b-adapt --no-b-pyramid

720×576p --ref 17 --bframes 3 --b-adapt --b-pyramid
720×576p --ref 19 --bframes 3 --b-adapt --no-b-pyramid

720×480p --ref 21 --bframes 3 --b-adapt --b-pyramid
720×480p --ref 23 --bframes 3 --b-adapt --no-b-pyramid

[Manao]

OT : CruNcher, don't forget your 2007 good resolution : make short and understandable sentences ;p

[CruNcher]

tetsuo55 i can only test this on the G92 Decoder Core for the moment later time also UVD, it would be nice if other would also test it on Xbox360/PS3 and if possible HD-DVD devices and Sony PSP i think if we get interoperability between those devices we have achived a big thing :)

[tetsuo55]

My calculations are wrong,

here is a better list:

http://www.avsforum.com/avs-vb/showpost.php?p=12704376&postcount=3898

Indeed we need a (lot) of help, maybe you could encode small sample files of all types, and it should fit on a single layer DVD, that way all the hardware players (except PSP) can be tested with a single disc. Ofcourse the files should be in a container that all the devices understand.

Testers would give feedback on which files do or do not work.

[UsedUser]

tetsuo55 i can only test this on the G92 Decoder Core for the moment later time also UVD, it would be nice if other would also test it on Xbox360/PS3 and if possible HD-DVD devices and Sony PSP i think if we get interoperability between those devices we have achived a big thing :)
My previous tests (http://forum.doom9.org/showthread.php?p=1081891#post1081891) demonstrated that the L4.1-compliant settings worked for 1080p and 720p on the ATI HD2600 (UVD chip). I also tested some in-between resolutions to ensure it is a DPB size limitation, and it proved to be true when 1920x864p worked with num_ref_frames = 5. I have since updated the Cyberlink decoder to the latest PowerDVD release (7.3.3516) on my box with the Nvidia 8800GT, and all of the L4.1-compliant files then worked with DXVA.

I have not tested 480p or 576p.

[tetsuo55]

UsedUser and CruNcher, What settings and players do you use to have DXVA in mkv files? and ar eyou guys using XP(i am)?

[CruNcher]

Yes windows XP Cyberlinks Decoder and Mplayer Home Cinema Classic for now VMR9 Windowed but i changed from a 7600 GS and still have to check the render changes (if their any) before the nicest was VMR9 Renderless running in the 3D Exclusive Fullscreen mode (ZoomPlayer has this too and can change dynamicly between it and Windowed) it gave the lesst jitter and absolute smooth playback with full Hardware Accelleration of VP1 :). But useing Haalis Splitter without changeing the Level_idc isn't recomended as it takes it into account and can then couse the aformentioned 20 fps or frame stuttering and in the evilst case Black Screen problems (Black Screen only with the Standard renderer (VMR7) with VMR9 you gonna see the first frame and no playback).
Best overall Expierience i currently make with Cyberlinks Player itself, but only because of the fact that it can play DVD,HD-DVD and Blu-Ray has no Menu Problems (there are some HD-DVD Menu problems with ACA content still but they slowly fix them) no Subtitle Problems (some other Player can't hold Hardware Accelleration when displaying subtitles) and all Accellerated and with Nvidias superb Motion Adaptive Deinterlacing if needed. So full Realtime 1920x1080 30i->1920x1080 60p conversion in a Quality that only the best Avisynth stuff can reach nowdays but that much much much much faster it's 1 step better quality then Yadif, so yes you could say it's MCBOB with NNEDI in Realtime (not quiet at least i didn't compared it yet against it or ATIs Deinterlacer) and that all together with the posibility to play most File Formats (and even HD-DVD and Blu-rays for testing from HDD little modification here ;) ) , where i still working on to get it to play even more, as soon as i achived that it will be my prefered player for this time being under Windows ;)

Im currently testing the 8800GT G92 (interoperability HD-DVD, Blu-Ray and DVB-S2 playback) already tested it's Gaming capabilities later im gonna test the new RV670 and finaly compare both against each other :) all the results of this you can read here http://www.forum-3dcenter.org/vbulletin/showthread.php?p=6152250, it's in German im constantly updateing it with results in all areas.

[akupenguin]

The new option needed is something to specify DPB size, and just let each frame use as many refs as possible within that constraint. Or I can repurpose --ref to do that: the current method was chosen because direct=temporal benefits from having the same number of L0 refs in all frames, but since direct=spatial is usually better anyway, it's no longer necessary.
patch (http://akuvian.org/src/x264/x264_ref_dpb.0.diff). That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

[foxyshadis]

You are probably right that they (accidentaly?) never go over the ref frames limit for the used resolutions

Cruncher did say that there were some block errors. What this means is that the high reference frames are being dropped and blocks referencing them broken, but decoders are much more robust than their anal level-checking gives them credit for. Given the relative distribution of high-ref frames, you could probably relevel even a 16-ref 1080p file and still see only mild artifacting.

[UsedUser]

UsedUser and CruNcher, What settings and players do you use to have DXVA in mkv files? and ar eyou guys using XP(i am)?
WinXP, ZoomPlayer using Overlay, MPC HC using VMR9 with 3D surfaces, Cyberlink decoder with Haali splitter. One box had an ATI HD2600 (and I'm currently waiting for a replacement) and another has an Nvidia 8800GT.

[UsedUser]

But useing Haalis Splitter without changeing the Level_idc isn't recomended as it takes it into account and can then couse the aformentioned 20 fps or frame stuttering and in the evilst case Black Screen problems (Black Screen only with the Standard renderer (VMR7) with VMR9 you gonna see the first frame and no playback).
It's not Haali's splitter at fault!

The level_idc is (also) checked by Cyberlink's decoder, as the raw AVC streams without a container exhibit the same stutter symptoms when level_idc is set higher than L4.1, and the same black screen when num_ref_frames is higher than L4.1.

[UsedUser]

No, that's what it already has. The new option needed is something to specify DPB size, and just let each frame use as many refs as possible within that constraint. Or I can repurpose --ref to do that: the current method was chosen because direct=temporal benefits from having the same number of L0 refs in all frames, but since direct=spatial is usually better anyway, it's no longer necessary.

patch (http://akuvian.org/src/x264/x264_ref_dpb.0.diff). That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?

Question:

According to your first post, I thought --ref would set DPB size, but obviously with --ref 1, it's not doing that with your patch. Is --ref 1 a special case?

[CruNcher]

patch (http://akuvian.org/src/x264/x264_ref_dpb.0.diff). That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

yep it works now
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

those 2 didn't work before the dbp 5 showed heavy frame errors and the dbp 7 didnt play @ all, now both work with dbp being 4 :)

Aku what do you think about a new parameter something like --hardware --sapc --forcesapc --hdc wich if used then forces num_ref_frame for the different resolutions :) this way then GUI devs could easily use it when the user want's a more strict Level of Hardware Decoder Compatibility (i think it would even make sense to bundle pictiming,aud,globalheader and hrd under this parameter) :) based on this list maybe ? http://www.avsforum.com/avs-vb/showpost.php?p=12704376&postcount=3898 or a even a easier way just allways allow 4 as maximum if --hdc is set, so if someone is so crazy and does --ref 15 if --hdc is set just ignore it and set it to 4 instead :d *hides*


I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?


Here is the Patched Build http://rapidshare.com/files/81854634/x264-dpb.exe

[CruNcher]

It's not Haali's splitter at fault!

The level_idc is (also) checked by Cyberlink's decoder, as the raw AVC streams without a container exhibit the same stutter symptoms when level_idc is set higher than L4.1, and the same black screen when num_ref_frames is higher than L4.1.

Yes but Level is not of importance try 5.1 in Bitstream in Cyberlink it will play then try that with Haalis Splitter and Cyberlink Decoder (MP4/MKV) and it won't play @ all with level_idc being 5.1.
Cyberlinks Framework doesn't check for the level_idc at least it doesn't result in a black screen like with Haalis Splitter if the Bitstream signals 5.1 but yes it indeed still shows stuttering (B-frame sequences) if the Level is wrong i wrote that some pages up allready (aku wished me even good luck finding the cause of this).
So the Black Screen with Haalis Splitter could also have another source im not sure the only thing i know is that black screens even with wrong signaled Bitstreams (5.1) can be avoided useing Mainconcepts Mp4 Demultiplexer and Cyberlinks Decoder with "Annex B output for AVC/H.264" checked, this input Cyberlinks Decoder accepts even if the Bitstreams level_idc is 5.1 it doesn't show a Black Screen in this combination, but it does with Haalis Splitter if the level_idc is 5.1 and it's not of interest what the num_ref_frame is level 5.1 is enough to cause a black screen.

So for everyone now, if the level_idc flag of a encoded Bitstream is 5.1 and you mux that into MP4 or MKV and try to playback it useing Haalis Splitter in combination with Cyberlinks Decoder with Hardware Accelleration Enabled the following will happen with VMR7 you gonna see a Black Screen with VMR9 you gonna see the first frame of the file and for both you would just see the slider moving but the time stays on 000000 so no Playback is happening, i hope now you understand what i mean.

[UsedUser]

Yes but Level is not of importance try 5.1 in Bitstream in Cyberlink it will play then try that with Haalis Splitter and Cyberlink Decoder (MP4/MKV) and it won't play @ all with level_idc being 5.1.

...

So for everyone now, if the level_idc flag of a encoded Bitstream is 5.1 and you mux that into MP4 or MKV and try to playback it useing Haalis Splitter in combination with Cyberlinks Decoder with Hardware Accelleration Enabled the following will happen with VMR7 you gonna see a Black Screen with VMR9 you gonna see the first frame of the file and for both you would just see the slider moving but the time stays on 000000 so no Playback is happening, i hope now you understand what i mean.
No, that's not what I've found at all.

As I've been saying over and over, the actual level of the stream determines DXVA playback, and the level_idc value determines smooth playback. I have found this is true no matter the container (no container, MP4, MKV), no matter the splitter (Haali's, Nero's, no splitter for raw AVC), and no matter the video renderer (VMR7, VMR9 Windowless/Renderless, Overlay).

There are three scenarios:

Stream is L5.1 (exceeds L4.1 DPB size), level_idc = L5.1
Result: No DXVA, black or gray screen

Stream is L4.1 (within L4.1 DPB size), level_idc = L5.1
Result: DXVA, but stuttering or 20fps playback

Stream is L4.1 (within L4.1 DPB size), level_idc = L4.1
Result: DXVA and smooth playback

[UsedUser]

I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?

Question:

According to your first post, I thought --ref would set DPB size, but obviously with --ref 1, it's not doing that with your patch. Is --ref 1 a special case?
I got my build environment updated and compiled my own build with the DPB patch.

As CruNcher confirmed, it appears the patch works as intended.

I tested the following @ 1080p:

-r4 -b#: dpb=4
-r4 -b# --b-pyramid: dpb=4

DXVA worked for both, but I saw block corruption with B-pyramids.

I also tested the following @ 720p:

-r9 -b#: dpb=9
-r9 -b# --b-pyramid: dpb=4

DXVA worked, but I saw block errors with both. If what foxyshadis said is true, having a DPB size close to the limit is causing the references to be dropped.

I went back and looked at the 720p clips I tested with the svn-709 build as well, and those clips with DPB=9 show block errors, too. The clips are:

-r6 -b# --b-pyramid: dpb=9
-r8 -b#: dpb=9
-r9 -b0: dpb=9

The clips that don't show any errors:

-r6 -b#: dpb=7
-r7 -b#: dpb=8


So, for whatever reason, to ensure good DXVA compatibility, it's still best to stay away from B-pyramids, and to keep DPB<5 for any resolution.

[UsedUser]

I tested clips at 480p and 576p on my Nvidia 8800GT (G92 decoder).

They worked with DXVA at L4.1, L4.0, L3.2 settings, but showed choppy playback and/or block errors.

My test methodology was to keep the DPB size within L4.1, which at either 480p or 576p allows DPB=16. Then I dropped the level_idc tag from 4.1 to 4.0 to 3.2 to 3.1, looking to see if the L4.1 streams could be decoded smoothly if the level_idc was lower. This demonstrated that the stream itself has to have DPB size within L3.1.

I then encoded 480p and 576p streams at their L3.1 DPB size limits, with the level_idc set at 4.1. DXVA worked, but as with the L4.1 streams with a level_idc of 5.1, playback was choppy with the level_idc set too high. This showed the level_idc needs to be L3.1.

For 480p and 576p, the DPB size needs to stay within L3.1, and the level_idc needs to be L3.1 for smooth playback.

The bottom line overall seems to be that streams need to comply with and declare L4.1 for HD and L3.1 for SD.

The following SD streams had smooth playback with DXVA:

480p, ref=13, bframes=3, DPB=13, level_idc=3.1
576p, ref=11, bframes=3, DPB=11, level_idc=3.1


The following SD streams had choppy playback with DXVA:

480p, ref=16, bframes=16, b-pyramid=on, DPB=16, level_idc=4.1
480p, ref=16, bframes=16, DPB=16, level_idc=4.1
480p, ref=16, bframes=16, DPB=16, level_idc=4.0
480p, ref=16, bframes=16, DPB=16, level_idc=3.2
480p, ref=16, bframes=16, DPB=16, level_idc=3.1
480p, ref=13, bframes=3, DPB=13, level_idc=4.1

576p, ref=16, bframes=16, b-pyramid=on, DPB=16, level_idc=4.1
576p, ref=16, bframes=16, DPB=16, level_idc=4.1
576p, ref=16, bframes=16, DPB=16, level_idc=4.0
576p, ref=16, bframes=16, DPB=16, level_idc=3.2
576p, ref=16, bframes=16, DPB=16, level_idc=3.1
576p, ref=11, bframes=3, DPB=11, level_idc=4.1

[akupenguin]

DXVA worked for both, but I saw block corruption with B-pyramids.
Next question: what is it about pyramid that your card didn't like?

Experiment 1: Conventional frame order but with B-refs. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_all-bref.diff) and encode without pyramid.

Experiment 2: Pyramid order but no B-refs. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_no-bref.diff) and encode with pyramid.

Experiment 3: Only a single L1 ref. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_no-L1.diff) and encode with pyramid.

All patches should be applied on top of the DPB patch (but not on top of each other).

[UsedUser]

Experiment 1: Conventional frame order but with B-refs. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_all-bref.diff) and encode without pyramid.
FYI - this patch compiled successfully, but the encoder crashes on the first pass. I don't have the time to look into it immediately.

Experiment 2: Pyramid order but no B-refs. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_no-bref.diff) and encode with pyramid.
Update: 1080p has no block errors with DPB=4; 720p continues to have block errors with DPB=9.


Experiment 3: Only a single L1 ref. Apply this patch (http://akuvian.org/src/x264/x264_dxva_pyramid_experiment_no-L1.diff) and encode with pyramid.
1080p may have block errors with DPB=4. It's difficult to tell if it's just visible macroblocking because it was difficult to encode, as the blocks only appear in one portion of the clip. I need to test some other clips.
Update: After more testing, I have determined it was not block errors I saw, but macroblocking on a difficult-to-encode segment. Other clips appear ok. Maybe macroblocking and block errors are essentially the same thing, but when I see errors, it doesn't look like regular, old "pixelation".

720p has block errors with DPB=9.

[CruNcher]

I got my build environment updated and compiled my own build with the DPB patch.

As CruNcher confirmed, it appears the patch works as intended.

I tested the following @ 1080p:

-r4 -b#: dpb=4
-r4 -b# --b-pyramid: dpb=4

DXVA worked for both, but I saw block corruption with B-pyramids.

I also tested the following @ 720p:

-r9 -b#: dpb=9
-r9 -b# --b-pyramid: dpb=4

DXVA worked, but I saw block errors with both. If what foxyshadis said is true, having a DPB size close to the limit is causing the references to be dropped.

I went back and looked at the 720p clips I tested with the svn-709 build as well, and those clips with DPB=9 show block errors, too. The clips are:

-r6 -b# --b-pyramid: dpb=9
-r8 -b#: dpb=9
-r9 -b0: dpb=9

The clips that don't show any errors:

-r6 -b#: dpb=7
-r7 -b#: dpb=8


So, for whatever reason, to ensure good DXVA compatibility, it's still best to stay away from B-pyramids, and to keep DPB<5 for any resolution.

i tested the 1 patch with CRF so far and i had no problems with b-pyramid at all in 1920x1080, strange i try more different combinations later on block errors i only had so far if the num_ref_frame wasn't identical with the encoded value and frame errors only if the level_idc flag was to low and the complexity of the stream to high for the signaled level.
i try 720p later on and yes you right it's not haalis splitter i checked it again it seems to be as you said a combination of both wrong level for wrong num_ref_frame :)

[akupenguin]

FYI - this patch compiled successfully, but the encoder crashes on the first pass. I don't have the time to look into it immediately.
Oops, you need to encode with at least as many refs as B-frames, otherwise some of the B-frames won't have any future ref.

I'm also interested in how the 3 experiments fare when not stressing the DPB. So try each with "-r2 -b2" and "-r3 -b3".

[UsedUser]

Oops, you need to encode with at least as many refs as B-frames, otherwise some of the B-frames won't have any future ref.
Hmm... I am using more refs than b-frames. Just checked again to verify, and the encoder still crashes. I'm using the following without b-pyramid:

1080p:
--ref 4 --bframes 3 --b-adapt

720p:
--ref 9 --bframes 3 --b-adapt


Using equal amounts or more b-frames than refs also crashed.

[Nil Einne]

What's the point of encoding cartoons with 16ref and not be able to watch them on PS3/X360 or PC with hardware acceleration? In this case I choose compatibility over small increase in quality.

Um if you don't have a PS2/X360 and don't intend to play in one, why would you give a damn what they can or can't do?

[CruNcher]

@Nil Einne
Interoperability allways comes with a drawback, but it allways should be prefered to keep it, this way you can avoid alot of problems in General, especialy for stuff that is based on a Standard and alot of people just wan't (includeing myself) that you have 1 Bitstream that plays everywhere if it's Standard Complaint (no transcoding no anything just take the bitstream and go wherever you want) :). The Next Problem for example is Apples Quicktime it's allready a defacto Standard itself (aside of the Real AVC/H.264 Standard) and it uses much less complexity (at least what Quicktime Player supports and so many are forced to support) and so we have another Problem keeping the Bitstream interoperable with that (sure it's easier to install Mplayer or VLC on a Mac and forget about that Problems, but sadly that's not the way it works), all this is really a Devils Circle. So even people like me that love interoperability need to make decissions, but at least you should try to hold them as low as possible and so HD-DVD and Blu-Ray seem a perfect balanced solution forgeting about Apple Quicktime wich can't deliver that and is really only usable for low complexity encoding (and seeing the rapid development of Hardware solutions it makes no sense anymore to stay that low complex all the time and loseing efficiency). But even if it sounds that easy it isn't that easy. Open Source can provide better Standard Complaint solutions (if some Companies don't want it or doing it wrong) for the User and so improve interoperability that way, but makeing that known to the users i think is the main problem here (and to deliver it in the easiest way possible). Implementing this what we talk about here now successfully means X264 is gonna reach more happy users in the end and for sure it also means alot more attention from different angles :). And exactly those Problems are those that give WMV and Microsoft VC-1 more and more momentum currently and Apple for example really does the wrong thing in many users eyes, but how much quality really goes lost with Apples setting ? and how Visualy relizeable ist that ?, those are questions i try to answer myself currently also. I think a dramaticly change of that Situation is gonna arrive if DivX Inc is gonna present DivX Q together with Mainconcept/Elecard that is gonna to get a new guide for many in the industry and will push H.264/AVC far ahead in fron't of WMV and finaly also force Apple to react (we could also achive the same now, it doesn't need to be a company as DivX Inc to be behind this, never forget where the roots of this company are). But it shouldn't be the Goal in the end to achive that with H.264/AVC (XviD 2.0 AVC or X264) we have to push other stuff and forget about those properiartary things (it's a nice thing to excercise and learn from it anyways in every possible perspective) :)

The nice thing is we have a battle going on here mostly between Microsoft (VC-1) and Sony (AVC) and when 2 parties have a War it's easier for a 3rd party to enter the Stage and get momentum (with more inovative stuff that the user are gonna use and love and then it doesn't matter where that's coming from) ;)