x264 Known Hardware accelleration problems and solutions

[UsedUser]

Originally Posted by CruNcher

tetsuo55 i can only test this on the G92 Decoder Core for the moment later time also UVD, it would be nice if other would also test it on Xbox360/PS3 and if possible HD-DVD devices and Sony PSP i think if we get interoperability between those devices we have achived a big thing

My previous tests demonstrated that the L4.1-compliant settings worked for 1080p and 720p on the ATI HD2600 (UVD chip). I also tested some in-between resolutions to ensure it is a DPB size limitation, and it proved to be true when 1920x864p worked with num_ref_frames = 5. I have since updated the Cyberlink decoder to the latest PowerDVD release (7.3.3516) on my box with the Nvidia 8800GT, and all of the L4.1-compliant files then worked with DXVA.

I have not tested 480p or 576p.

[tetsuo55]

UsedUser and CruNcher, What settings and players do you use to have DXVA in mkv files? and ar eyou guys using XP(i am)?

[CruNcher]

Yes windows XP Cyberlinks Decoder and Mplayer Home Cinema Classic for now VMR9 Windowed but i changed from a 7600 GS and still have to check the render changes (if their any) before the nicest was VMR9 Renderless running in the 3D Exclusive Fullscreen mode (ZoomPlayer has this too and can change dynamicly between it and Windowed) it gave the lesst jitter and absolute smooth playback with full Hardware Accelleration of VP1 . But useing Haalis Splitter without changeing the Level_idc isn't recomended as it takes it into account and can then couse the aformentioned 20 fps or frame stuttering and in the evilst case Black Screen problems (Black Screen only with the Standard renderer (VMR7) with VMR9 you gonna see the first frame and no playback).
Best overall Expierience i currently make with Cyberlinks Player itself, but only because of the fact that it can play DVD,HD-DVD and Blu-Ray has no Menu Problems (there are some HD-DVD Menu problems with ACA content still but they slowly fix them) no Subtitle Problems (some other Player can't hold Hardware Accelleration when displaying subtitles) and all Accellerated and with Nvidias superb Motion Adaptive Deinterlacing if needed. So full Realtime 1920x1080 30i->1920x1080 60p conversion in a Quality that only the best Avisynth stuff can reach nowdays but that much much much much faster it's 1 step better quality then Yadif, so yes you could say it's MCBOB with NNEDI in Realtime (not quiet at least i didn't compared it yet against it or ATIs Deinterlacer) and that all together with the posibility to play most File Formats (and even HD-DVD and Blu-rays for testing from HDD little modification here ) , where i still working on to get it to play even more, as soon as i achived that it will be my prefered player for this time being under Windows

Im currently testing the 8800GT G92 (interoperability HD-DVD, Blu-Ray and DVB-S2 playback) already tested it's Gaming capabilities later im gonna test the new RV670 and finaly compare both against each other all the results of this you can read here http://www.forum-3dcenter.org/vbulle....php?p=6152250, it's in German im constantly updateing it with results in all areas.

[akupenguin]

Originally Posted by akupenguin

The new option needed is something to specify DPB size, and just let each frame use as many refs as possible within that constraint. Or I can repurpose --ref to do that: the current method was chosen because direct=temporal benefits from having the same number of L0 refs in all frames, but since direct=spatial is usually better anyway, it's no longer necessary.

patch. That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

[foxyshadis]

Originally Posted by tetsuo55

You are probably right that they (accidentaly?) never go over the ref frames limit for the used resolutions

Cruncher did say that there were some block errors. What this means is that the high reference frames are being dropped and blocks referencing them broken, but decoders are much more robust than their anal level-checking gives them credit for. Given the relative distribution of high-ref frames, you could probably relevel even a 16-ref 1080p file and still see only mild artifacting.

[UsedUser]

Originally Posted by tetsuo55

UsedUser and CruNcher, What settings and players do you use to have DXVA in mkv files? and ar eyou guys using XP(i am)?

WinXP, ZoomPlayer using Overlay, MPC HC using VMR9 with 3D surfaces, Cyberlink decoder with Haali splitter. One box had an ATI HD2600 (and I'm currently waiting for a replacement) and another has an Nvidia 8800GT.

[UsedUser]

Originally Posted by CruNcher

But useing Haalis Splitter without changeing the Level_idc isn't recomended as it takes it into account and can then couse the aformentioned 20 fps or frame stuttering and in the evilst case Black Screen problems (Black Screen only with the Standard renderer (VMR7) with VMR9 you gonna see the first frame and no playback).

It's not Haali's splitter at fault!

The level_idc is (also) checked by Cyberlink's decoder, as the raw AVC streams without a container exhibit the same stutter symptoms when level_idc is set higher than L4.1, and the same black screen when num_ref_frames is higher than L4.1.

[UsedUser]

Originally Posted by akupenguin

No, that's what it already has. The new option needed is something to specify DPB size, and just let each frame use as many refs as possible within that constraint. Or I can repurpose --ref to do that: the current method was chosen because direct=temporal benefits from having the same number of L0 refs in all frames, but since direct=spatial is usually better anyway, it's no longer necessary.

Originally Posted by akupenguin

patch. That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?

Question:

According to your first post, I thought --ref would set DPB size, but obviously with --ref 1, it's not doing that with your patch. Is --ref 1 a special case?

[CruNcher]

Originally Posted by akupenguin

patch. That was easy, now tell me whether it worked.

What should happen (untested):

before:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r1 -b# --b-pyramid: dpb=4, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

after:
-r1 -b0: dpb=1, P-frames use 1 refs
-r4 -b0: dpb=4, P-frames use 4 refs
-r1 -b#: dpb=2, P-frames use 1 refs, B-frames use 1/1 refs
-r4 -b#: dpb=4, P-frames use 4 refs, B-frames use 3/1 refs
-r1 -b# --b-pyramid: dpb=3, P-frames use 1 refs, B-refs use 1/1 refs, B-frames use 1/2 refs
-r4 -b# --b-pyramid: dpb=4, P-frames use 4 refs, B-refs use 3/1 refs, B-frames use 2/2 refs

yep it works now
-r4 -b#: dpb=5, P-frames use 4 refs, B-frames use 4/1 refs
-r4 -b# --b-pyramid: dpb=7, P-frames use 4 refs, B-refs use 4/1 refs, B-frames use 4/2 refs

those 2 didn't work before the dbp 5 showed heavy frame errors and the dbp 7 didnt play @ all, now both work with dbp being 4

Aku what do you think about a new parameter something like --hardware --sapc --forcesapc --hdc wich if used then forces num_ref_frame for the different resolutions this way then GUI devs could easily use it when the user want's a more strict Level of Hardware Decoder Compatibility (i think it would even make sense to bundle pictiming,aud,globalheader and hrd under this parameter) based on this list maybe ? http://www.avsforum.com/avs-vb/showp...postcount=3898 or a even a easier way just allways allow 4 as maximum if --hdc is set, so if someone is so crazy and does --ref 15 if --hdc is set just ignore it and set it to 4 instead :d *hides*

I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?

Here is the Patched Build http://rapidshare.com/files/81854634/x264-dpb.exe

[CruNcher]

Originally Posted by UsedUser

It's not Haali's splitter at fault!

The level_idc is (also) checked by Cyberlink's decoder, as the raw AVC streams without a container exhibit the same stutter symptoms when level_idc is set higher than L4.1, and the same black screen when num_ref_frames is higher than L4.1.

Yes but Level is not of importance try 5.1 in Bitstream in Cyberlink it will play then try that with Haalis Splitter and Cyberlink Decoder (MP4/MKV) and it won't play @ all with level_idc being 5.1.
Cyberlinks Framework doesn't check for the level_idc at least it doesn't result in a black screen like with Haalis Splitter if the Bitstream signals 5.1 but yes it indeed still shows stuttering (B-frame sequences) if the Level is wrong i wrote that some pages up allready (aku wished me even good luck finding the cause of this).
So the Black Screen with Haalis Splitter could also have another source im not sure the only thing i know is that black screens even with wrong signaled Bitstreams (5.1) can be avoided useing Mainconcepts Mp4 Demultiplexer and Cyberlinks Decoder with "Annex B output for AVC/H.264" checked, this input Cyberlinks Decoder accepts even if the Bitstreams level_idc is 5.1 it doesn't show a Black Screen in this combination, but it does with Haalis Splitter if the level_idc is 5.1 and it's not of interest what the num_ref_frame is level 5.1 is enough to cause a black screen.

So for everyone now, if the level_idc flag of a encoded Bitstream is 5.1 and you mux that into MP4 or MKV and try to playback it useing Haalis Splitter in combination with Cyberlinks Decoder with Hardware Accelleration Enabled the following will happen with VMR7 you gonna see a Black Screen with VMR9 you gonna see the first frame of the file and for both you would just see the slider moving but the time stays on 000000 so no Playback is happening, i hope now you understand what i mean.

[UsedUser]

Originally Posted by CruNcher

Yes but Level is not of importance try 5.1 in Bitstream in Cyberlink it will play then try that with Haalis Splitter and Cyberlink Decoder (MP4/MKV) and it won't play @ all with level_idc being 5.1.

...

So for everyone now, if the level_idc flag of a encoded Bitstream is 5.1 and you mux that into MP4 or MKV and try to playback it useing Haalis Splitter in combination with Cyberlinks Decoder with Hardware Accelleration Enabled the following will happen with VMR7 you gonna see a Black Screen with VMR9 you gonna see the first frame of the file and for both you would just see the slider moving but the time stays on 000000 so no Playback is happening, i hope now you understand what i mean.

No, that's not what I've found at all.

As I've been saying over and over, the actual level of the stream determines DXVA playback, and the level_idc value determines smooth playback. I have found this is true no matter the container (no container, MP4, MKV), no matter the splitter (Haali's, Nero's, no splitter for raw AVC), and no matter the video renderer (VMR7, VMR9 Windowless/Renderless, Overlay).

There are three scenarios:

Stream is L5.1 (exceeds L4.1 DPB size), level_idc = L5.1
Result: No DXVA, black or gray screen

Stream is L4.1 (within L4.1 DPB size), level_idc = L5.1
Result: DXVA, but stuttering or 20fps playback

Stream is L4.1 (within L4.1 DPB size), level_idc = L4.1
Result: DXVA and smooth playback

[UsedUser]

Originally Posted by UsedUser

I'll have to wait until a build is available to test. I'm not set up to build. Anyone able to do this?

Question:

According to your first post, I thought --ref would set DPB size, but obviously with --ref 1, it's not doing that with your patch. Is --ref 1 a special case?

I got my build environment updated and compiled my own build with the DPB patch.

As CruNcher confirmed, it appears the patch works as intended.

I tested the following @ 1080p:

-r4 -b#: dpb=4
-r4 -b# --b-pyramid: dpb=4

DXVA worked for both, but I saw block corruption with B-pyramids.

I also tested the following @ 720p:

-r9 -b#: dpb=9
-r9 -b# --b-pyramid: dpb=4

DXVA worked, but I saw block errors with both. If what foxyshadis said is true, having a DPB size close to the limit is causing the references to be dropped.

I went back and looked at the 720p clips I tested with the svn-709 build as well, and those clips with DPB=9 show block errors, too. The clips are:

-r6 -b# --b-pyramid: dpb=9
-r8 -b#: dpb=9
-r9 -b0: dpb=9

The clips that don't show any errors:

-r6 -b#: dpb=7
-r7 -b#: dpb=8


So, for whatever reason, to ensure good DXVA compatibility, it's still best to stay away from B-pyramids, and to keep DPB<5 for any resolution.

[UsedUser]

I tested clips at 480p and 576p on my Nvidia 8800GT (G92 decoder).

They worked with DXVA at L4.1, L4.0, L3.2 settings, but showed choppy playback and/or block errors.

My test methodology was to keep the DPB size within L4.1, which at either 480p or 576p allows DPB=16. Then I dropped the level_idc tag from 4.1 to 4.0 to 3.2 to 3.1, looking to see if the L4.1 streams could be decoded smoothly if the level_idc was lower. This demonstrated that the stream itself has to have DPB size within L3.1.

I then encoded 480p and 576p streams at their L3.1 DPB size limits, with the level_idc set at 4.1. DXVA worked, but as with the L4.1 streams with a level_idc of 5.1, playback was choppy with the level_idc set too high. This showed the level_idc needs to be L3.1.

For 480p and 576p, the DPB size needs to stay within L3.1, and the level_idc needs to be L3.1 for smooth playback.

The bottom line overall seems to be that streams need to comply with and declare L4.1 for HD and L3.1 for SD.

The following SD streams had smooth playback with DXVA:

480p, ref=13, bframes=3, DPB=13, level_idc=3.1
576p, ref=11, bframes=3, DPB=11, level_idc=3.1


The following SD streams had choppy playback with DXVA:

480p, ref=16, bframes=16, b-pyramid=on, DPB=16, level_idc=4.1
480p, ref=16, bframes=16, DPB=16, level_idc=4.1
480p, ref=16, bframes=16, DPB=16, level_idc=4.0
480p, ref=16, bframes=16, DPB=16, level_idc=3.2
480p, ref=16, bframes=16, DPB=16, level_idc=3.1
480p, ref=13, bframes=3, DPB=13, level_idc=4.1

576p, ref=16, bframes=16, b-pyramid=on, DPB=16, level_idc=4.1
576p, ref=16, bframes=16, DPB=16, level_idc=4.1
576p, ref=16, bframes=16, DPB=16, level_idc=4.0
576p, ref=16, bframes=16, DPB=16, level_idc=3.2
576p, ref=16, bframes=16, DPB=16, level_idc=3.1
576p, ref=11, bframes=3, DPB=11, level_idc=4.1

[akupenguin]

Originally Posted by UsedUser

DXVA worked for both, but I saw block corruption with B-pyramids.

Next question: what is it about pyramid that your card didn't like?

Experiment 1: Conventional frame order but with B-refs. Apply this patch and encode without pyramid.

Experiment 2: Pyramid order but no B-refs. Apply this patch and encode with pyramid.

Experiment 3: Only a single L1 ref. Apply this patch and encode with pyramid.

All patches should be applied on top of the DPB patch (but not on top of each other).

[UsedUser]

Originally Posted by akupenguin

Experiment 1: Conventional frame order but with B-refs. Apply this patch and encode without pyramid.

FYI - this patch compiled successfully, but the encoder crashes on the first pass. I don't have the time to look into it immediately.

Originally Posted by akupenguin

Experiment 2: Pyramid order but no B-refs. Apply this patch and encode with pyramid.

Update: 1080p has no block errors with DPB=4; 720p continues to have block errors with DPB=9.

Originally Posted by akupenguin

Experiment 3: Only a single L1 ref. Apply this patch and encode with pyramid.

1080p may have block errors with DPB=4. It's difficult to tell if it's just visible macroblocking because it was difficult to encode, as the blocks only appear in one portion of the clip. I need to test some other clips.
Update: After more testing, I have determined it was not block errors I saw, but macroblocking on a difficult-to-encode segment. Other clips appear ok. Maybe macroblocking and block errors are essentially the same thing, but when I see errors, it doesn't look like regular, old "pixelation".

720p has block errors with DPB=9.

[CruNcher]

Originally Posted by UsedUser

I got my build environment updated and compiled my own build with the DPB patch.

As CruNcher confirmed, it appears the patch works as intended.

I tested the following @ 1080p:

-r4 -b#: dpb=4
-r4 -b# --b-pyramid: dpb=4

DXVA worked for both, but I saw block corruption with B-pyramids.

I also tested the following @ 720p:

-r9 -b#: dpb=9
-r9 -b# --b-pyramid: dpb=4

DXVA worked, but I saw block errors with both. If what foxyshadis said is true, having a DPB size close to the limit is causing the references to be dropped.

I went back and looked at the 720p clips I tested with the svn-709 build as well, and those clips with DPB=9 show block errors, too. The clips are:

-r6 -b# --b-pyramid: dpb=9
-r8 -b#: dpb=9
-r9 -b0: dpb=9

The clips that don't show any errors:

-r6 -b#: dpb=7
-r7 -b#: dpb=8


So, for whatever reason, to ensure good DXVA compatibility, it's still best to stay away from B-pyramids, and to keep DPB<5 for any resolution.

i tested the 1 patch with CRF so far and i had no problems with b-pyramid at all in 1920x1080, strange i try more different combinations later on block errors i only had so far if the num_ref_frame wasn't identical with the encoded value and frame errors only if the level_idc flag was to low and the complexity of the stream to high for the signaled level.
i try 720p later on and yes you right it's not haalis splitter i checked it again it seems to be as you said a combination of both wrong level for wrong num_ref_frame

[akupenguin]

Originally Posted by UsedUser

FYI - this patch compiled successfully, but the encoder crashes on the first pass. I don't have the time to look into it immediately.

Oops, you need to encode with at least as many refs as B-frames, otherwise some of the B-frames won't have any future ref.

I'm also interested in how the 3 experiments fare when not stressing the DPB. So try each with "-r2 -b2" and "-r3 -b3".

[UsedUser]

Originally Posted by akupenguin

Oops, you need to encode with at least as many refs as B-frames, otherwise some of the B-frames won't have any future ref.

Hmm... I am using more refs than b-frames. Just checked again to verify, and the encoder still crashes. I'm using the following without b-pyramid:

1080p:
--ref 4 --bframes 3 --b-adapt

720p:
--ref 9 --bframes 3 --b-adapt


Using equal amounts or more b-frames than refs also crashed.

[Nil Einne]

Originally Posted by Atak_Snajpera

What's the point of encoding cartoons with 16ref and not be able to watch them on PS3/X360 or PC with hardware acceleration? In this case I choose compatibility over small increase in quality.

Um if you don't have a PS2/X360 and don't intend to play in one, why would you give a damn what they can or can't do?

[CruNcher]

@Nil Einne
Interoperability allways comes with a drawback, but it allways should be prefered to keep it, this way you can avoid alot of problems in General, especialy for stuff that is based on a Standard and alot of people just wan't (includeing myself) that you have 1 Bitstream that plays everywhere if it's Standard Complaint (no transcoding no anything just take the bitstream and go wherever you want) . The Next Problem for example is Apples Quicktime it's allready a defacto Standard itself (aside of the Real AVC/H.264 Standard) and it uses much less complexity (at least what Quicktime Player supports and so many are forced to support) and so we have another Problem keeping the Bitstream interoperable with that (sure it's easier to install Mplayer or VLC on a Mac and forget about that Problems, but sadly that's not the way it works), all this is really a Devils Circle. So even people like me that love interoperability need to make decissions, but at least you should try to hold them as low as possible and so HD-DVD and Blu-Ray seem a perfect balanced solution forgeting about Apple Quicktime wich can't deliver that and is really only usable for low complexity encoding (and seeing the rapid development of Hardware solutions it makes no sense anymore to stay that low complex all the time and loseing efficiency). But even if it sounds that easy it isn't that easy. Open Source can provide better Standard Complaint solutions (if some Companies don't want it or doing it wrong) for the User and so improve interoperability that way, but makeing that known to the users i think is the main problem here (and to deliver it in the easiest way possible). Implementing this what we talk about here now successfully means X264 is gonna reach more happy users in the end and for sure it also means alot more attention from different angles . And exactly those Problems are those that give WMV and Microsoft VC-1 more and more momentum currently and Apple for example really does the wrong thing in many users eyes, but how much quality really goes lost with Apples setting ? and how Visualy relizeable ist that ?, those are questions i try to answer myself currently also. I think a dramaticly change of that Situation is gonna arrive if DivX Inc is gonna present DivX Q together with Mainconcept/Elecard that is gonna to get a new guide for many in the industry and will push H.264/AVC far ahead in fron't of WMV and finaly also force Apple to react (we could also achive the same now, it doesn't need to be a company as DivX Inc to be behind this, never forget where the roots of this company are). But it shouldn't be the Goal in the end to achive that with H.264/AVC (XviD 2.0 AVC or X264) we have to push other stuff and forget about those properiartary things (it's a nice thing to excercise and learn from it anyways in every possible perspective)

The nice thing is we have a battle going on here mostly between Microsoft (VC-1) and Sony (AVC) and when 2 parties have a War it's easier for a 3rd party to enter the Stage and get momentum (with more inovative stuff that the user are gonna use and love and then it doesn't matter where that's coming from)