wide range of possible reference frames in mpeg-4 avc/h.264?

[bond]

blame on me if this has been discussed already, but i couldnt find it

i wonder about whether allowing a huge range of possible multiple reference frames would actually help compressibility
i mean in most movies there often is a switch between different scenes (eg if two people talk with each other), where eg first person A is shown talking (scene 1), than the picture switches to person B (scene 2), and than to person A again (scene 3, showing the same camera angle aso as in scene 1...)

couldnt multiple reference frames be very useful if they could be used between different scenes, meaning for example referencing from scene 3 to scene 1?
i realise that this would slow down encoding a lot, but couldnt this be optimised by also taking scenechanges in account (eg only looking at the frames shortly before a scenechange at the end of the scene before the last scene), realising that there are really a lot switches between/to similar scenes in movies? so to say it could be an option created which says: only look at the 3 last frames at the end of the 3 last scenechanges or so...

damn, its hard to describe what i mean, i hope its clear