x264 P-Frames not marked as reference frames

[flieger30]

For temporal scalability I need a stream like this (just an example, my stream is indeed a bit more complicated)

Example a)
..._....._...._....._.
../.\.../.\../.\.../.\
./\..\./\..\/\..\./\..\
I.P1.P2.P3.P4.P5.P6.P7.P8

or ( a little harder..)

Example b)
...._____......_____.
.../.....\..../.....\
../.\.....\../.\.....\
./\..\./\..\/\..\./\..\
I.P1.P2.P3.P4.P5.P6.P7.P8.


Let's focus on a)

Frame - Predicted from
P1 - I
P2 - I
P3 - P2
P4 - P2
P5 - P4
P6 - P4
P7 - P6
P8 - P6

This sheme needs num_ref_frames=1 -> IMPORTANT, the decoder needs only 1 ref frame!
As you see, P1,P3,P5,P7 are NOT used as references frames, they should be coded as P-Frames but with nal_ref_idc=0.
But when i set i_nal_ref_idc=0 in (encoder.c, line 2952)
i_nal_ref_idc = NAL_PRIORITY_DISPOSABLE;
// x264_reference_hierarchy_reset( h ); // removed for nal_ref_idc=0
h->sh.i_type = SLICE_TYPE_P;
the encoder is not working correct anymore.

So antoher approach is: I let the encoder work as normal, but AFTER encoding P1, I patch header in the decoded bitstream, where I set nal-ref_idc=0 for this frame and then I tell the encoder, to not use this frame anymore by calling x264_encoder_invalidate_reference(..)
But eigentlich i want to tell the encoder BEFORE coding P1, that this frame is NOT used as a reference frame (there is no need to waiste time in re-decode the encoded picture).

Case b)
Same problem as a.
As you see , I need num_ref_frame=3 (worst case is P8, which needs I,P4,P6 as refeerence frames).
But when I encode P7 (I,P4,P6 are in idb), and I can't tell the encoder BEFORE encoding P7, that nal_ref_idc should be 0 for this frame, he will throw away the I from idb and put P7 in, so that P4,P6 and P7 remain in idb. The work around to patch the encoded bitstreasm headers and adjust the picture buffer with x264_encoder_invalidate_reference(..) will only work with num_ref_frames=4 => i had to patch the num_ref_frames in the headers too.

Questions:
Is there a way to tell the encoder BEFORE encoding P-Frames, that he should use nal_ref_idc=0, without producing a corrupted stream?
Is the "patch headers and remove pictures manually after encoding" workaround the way to handle the things?

[Warperus]

There should be a way.
Nope, it's bloody hack

[akupenguin]

Why do you want to limit num_ref_frames?