wide range of possible reference frames in mpeg-4 avc/h.264?

[bond]

blame on me if this has been discussed already, but i couldnt find it

i wonder about whether allowing a huge range of possible multiple reference frames would actually help compressibility
i mean in most movies there often is a switch between different scenes (eg if two people talk with each other), where eg first person A is shown talking (scene 1), than the picture switches to person B (scene 2), and than to person A again (scene 3, showing the same camera angle aso as in scene 1...)

couldnt multiple reference frames be very useful if they could be used between different scenes, meaning for example referencing from scene 3 to scene 1?
i realise that this would slow down encoding a lot, but couldnt this be optimised by also taking scenechanges in account (eg only looking at the frames shortly before a scenechange at the end of the scene before the last scene), realising that there are really a lot switches between/to similar scenes in movies? so to say it could be an option created which says: only look at the 3 last frames at the end of the 3 last scenechanges or so...

damn, its hard to describe what i mean, i hope its clear :D

[Blue_MiSfit]

that's a really good idea IMHO :)

Although most ideas sound good to me, and quite often my own sound great and then don't when I tell other people about them...

Regardless, this would be interesting, but as you said VERY slow... Ateme's AVC really slows down above 1 or 2 of these, no?

[PowerMacG4]

When the I/P/B decision thinks there is enough change in a frame to warrant an I frame, it could check back to the 2 or so previous P-frames before the previous I-frames and see the deviation from those, and then use the smallest P frame instead. This would KILL seeking, though... I suppose.

[akupenguin]

Yes, it's doable. There is a provision in the spec for "long-term references", and I can't imagine what else they intended it for.
It wouldn't even be too slow if you restrict it to scenechanges.
And it would only kill seeking in a 1-pass encode; the second pass would know when to keep long-term refs, and when it's ok to insert seek-points.

So it's on my todo list for x264. But I'm not Ateme, and I do this in my spare time, so no guarantees on when I finish.

[Mug Funky]

i don't know how much would be gained from this though... i mean, what percentage of a movie's total size is made up of i-frames?

though one thing it'd be great for is shorter term stuff - think of all that news footage you see where there's a relatively boring and low motion scene (some fat politician walking around or shaking hands), that is sent straight to q31 hell because of camera flashes. multiple reference frames would handle these scenes with almost no increase in size, i would think. a similar situation exists with music videos.

also, wouldn't seeking be fine if the referred frames are only i-frames?
i suppose there'd be less increase in efficiency in a lot of cases though.

[edit]

Akupenguin: isn't anybody else working on x264? i'd do it but i can't code and don't have the maths, so i wouldn't be of much use :(

[Valeron]

Good idea, indeed.
But maybe not for AVC or any present generation codec.
When the CPU is fast enough, I think features include this complex idea will be add into the video compression tech.

btw, I think "Cell" can handle this but not P4.:D

[snacky]

Originally posted by Mug Funky

though one thing it'd be great for is shorter term stuff - think of all that news footage you see where there's a relatively boring and low motion scene (some fat politician walking around or shaking hands), that is sent straight to q31 hell because of camera flashes. multiple reference frames would handle these scenes with almost no increase in size, i would think.

The use of multiple reference frames DOES help here, and for a camera flash, it may only take two reference frames to drastically reduce the required bitrate.

For an admittedly unrealistic example of how multiple reference frames can help with flashing scenes, see http://ikaruga.co.uk/~snacky/rft/hq-version/

The first file was encoded with only one reference frame. The other ones all have identical video content, yet they're less than 7% the size because they use multiple references. (as you can see in this case, only 2 are needed and allowing more has no effect.) akupenguin tells me that even greater efficiency could be achieved if x264 implemented reference list reordering.

[akupenguin]

Originally posted by Mug Funky
i don't know how much would be gained from this though... i mean, what percentage of a movie's total size is made up of i-frames? I-frames are somewhere around 1% by framecount and 5% by byte size in my encodes. But that's normal movies, without too many camera flashes and the like.

though one thing it'd be great for is shorter term stuff - think of all that news footage you see where there's a relatively boring and low motion scene (some fat politician walking around or shaking hands), that is sent straight to q31 hell because of camera flashes. multiple reference frames would handle these scenes with almost no increase in size, i would think. a similar situation exists with music videos. Yes, multiple reference frames are great for that sort o thing. But x264 uses them suboptimally, due to the fixed IDR cycle. (Set idrint=999 to see how much compression you could potentially gain with the optimal decision algorithm. But that will kill seeking until I actually impelement such an algorithm.)

also, wouldn't seeking be fine if the referred frames are only i-frames?
i suppose there'd be less increase in efficiency in a lot of cases though. a) You probably want to keep the most recent frame in the previous scene, not the first frame.
b) It's not allowed. Decoding must be separate from seeking, and the decoder does not have callbacks into the demuxer. It could work if you design the whole media playback system around a new codec, but not in H.264.

Akupenguin: isn't anybody else working on x264? i'd do it but i can't code and don't have the maths, so i wouldn't be of much use :(
We get the occasional patch, and other devs are sometimes around to apply them, but I think no one else is doing much related to the compression algorithms.

[Gabriel_Bouvigne]

The first file was encoded with only one reference frame. The other ones all have identical video content, yet they're less than 7% the size because they use multiple references. (as you can see in this case, only 2 are needed and allowing more has no effect.)
Even with only 1 reference frame, you can still choose your reference frame.

[bond]

Originally posted by akupenguin
Yes, it's doable. There is a provision in the spec for "long-term references", and I can't imagine what else they intended it for.
It wouldn't even be too slow if you restrict it to scenechanges.
And it would only kill seeking in a 1-pass encode; the second pass would know when to keep long-term refs, and when it's ok to insert seek-points.

So it's on my todo list for x264. But I'm not Ateme, and I do this in my spare time, so no guarantees on when I finish.sounds great :)

is this part of the normal multiple reference specs (eg in the main profile) or is this part of some special extension (eg frext or so)?

[akupenguin]

It's part of Main profile, and libavcodec supports it. (At least, I see code with that intended result. As I don't know of any encoder that uses it, and JM is a pain to deal with, I haven't tested the support.)