How to get more b-frames? XviD not using my settings :(

[Zep]

How does Xvid decide how many b-frames in a row because it sure is
not using what i typed in :)

I set it to 4 b-frames but Xvid (RC4) never does more than about 1.5 to 1
average ratio to P frames no matter what i try. Even when I enter 2
the ratio is only 1.5 to 1 also.


I realize it says "Max consecutive BVOPS" and thus no guarantee/forcing
of that many but I want it forced to what i enter.

How can this be done?


Does anyone know How Xvid decides when to use 4 in a row and when/why
not to? For the heck of it I tried 100 b-frames and still the same result. 1.5 to 1 on a 42 minute encode.


thanks

[iradic]

http://www.vslcatena.nl/~ronald/docs/xvidfaq.html#C11

[Zep]

Originally posted by iradic
http://www.vslcatena.nl/~ronald/docs/xvidfaq.html#C11

hmmmm... i wish it said HOW it determines when to use the max
and WHY the default is set so low as to only get 1.5 (or lower)
to 1 ratio. I mean it states how great they are then doesn't
use their full potenial in default setttings. Which is strange.

I guess I will just max it to 25+ and force it to hit whatever I enter


That leads to a test i want to try.

force 4+ in row AND

force b-frames to Q=2 (or Q=3) since so many in a row will start
to degrade video if Q=4.

Since b-frames at Q=2 are 1/3 the size of P frames at Q=2 this
should work well on bit savings and still look great.


thx for the link

[sysKin]

Defaults are optimized for maximum PSNR. Since noone trusts PSNR, you have this extra sensitivity setting available.

However, be warned that your assumptions are pretty bad - bframes are smaller than p-frames *not* because they just are, but because of their position relative to p-frames. The more b-frames, the worse compression in all of them *and* the worse compression of p-frames as well.

You were given these settings so that you can experiment, do so - but don't say that defaults are bad unless you can really prove otherwise.

Radek

[Zep]

Originally posted by iradic
http://www.vslcatena.nl/~ronald/docs/xvidfaq.html#C11

grrrr.... i set the default zone and only zone to BVOP sensitivity to 25 and only 2.1 to 1 ratio.

geez lol


better but I guess 25 is not really the "practical maximum" :D


will try 100 next

[lordadmira]

Hmm, well I guess what ur really asking here is for some explanation on B frame theory. And honestly it's not really documented well anywhere, AFAIK. To understand the power of B frames u need to realize what type of frame sequences they can exploit. There are two basic types of motion in a video, progressive motion and periodic motion. And really, progressive motion is a special case of periodic motion where the number of periods is 1 or less. Think of a static scene, and a car drives across the scene and then out of the frame. This is progressive motion. The motion is going in one direction through time. Think of a scene where a person is walking toward the camera. There is no motion left or right but only in the temporal direction. This too is progressive motion and is best handled by P frames. This is because the image information for the moving object constantly requires new information, not information that has already been encoded(about the moving object). But from where can u predict the background information that was swept-over by the moving car? You can't. It has to be encoded *new* for each frame. If we reconsider this as a period 1 motion we can use B frames to *save* all that area that would otherwise have to be coded anew.

Now consider periodic motion. This is exactly what it sounds like. An object is moving in one direction and then moves back where it's previously been, or nearly so. A B frame will reuse information from both the beginning (like a P frame) and end of a motion. Think of a scene where a weatherman is in front of the weather map and it changes as he walks back and forth across it. With P frames alone the codec has to newly encode information about the background in each frame as he moves around. Only the (nearly)static portions will be carried forward. If you have B frames, you can predict those new areas from the future. One half of the B frame is anchored in the past, the peak of the period, and the other half is anchored in the future, the trough of the period so to speak. Where he's going is predicted from the past, and where he's been is predicted from the future. This effect is most profound in animation where they make heavy use of repeating loops of frames.

So how many B frames should I use? It depends on ur source material. The codec will try to lock on to those motion end-points encoding them as P frames to use B frames in between. The more B frames u call for, and the higher the sensitivity, the wider and more extensive the search for periodic motions. The farther into the future it will look. Generally, this is good. With experience u will just "know" the B frame potential for a particular video. The B frame "window" has to be large enough to encompass most of the motions in the video. The motion periods dictate the requisite B frame settings.

In my encodings I usually set B frames to 8/1/0, meaning max of 8 in a row and full quality. And sensitivity to 80. With this kind of a setup P frames become defacto I frames and the main load is carried by the B frames. Watching the Xvid status window, I can tell u that this does result in drastically smaller B frames of 4-6 in a row. Now I mainly encode anime so it might not be right for ur purposes. But I have gotten very good results with live material. So my advice: go for it. If it looks crappy then u have some strange material, try lowering sens and raising offset.

LA

[Pen-Pen]

Interesting settings indeed ! I never thought about going for a lot of consecutive and hi-q b-frames... That's something I'm gonna try :) I usually go for max 2 Q4-BF...

[Zep]

Originally posted by sysKin
Defaults are optimized for maximum PSNR. Since noone trusts PSNR, you have this extra sensitivity setting available.

However, be warned that your assumptions are pretty bad - bframes are smaller than p-frames *not* because they just are, but because of their position relative to p-frames. The more b-frames, the worse compression in all of them *and* the worse compression of p-frames as well.

You were given these settings so that you can experiment, do so - but don't say that defaults are bad unless you can really prove otherwise.

Radek


ahh ok NOW we are getting somewhere. :)

i tested against what you said


it took a sensitivity of 100 to hit the 4 to 1 I wanted.
tried 25 , 50 , 75 (75 was just under a 3.4 to 1)

I forced B-frames to just 1 Q above P frames instead of 2
to make up for the extended run of lower quality frames.

the lastest one was BY FAR the best.

i frames where about the same. same total and average size
give or take.

b-frames were only .5K bigger than sensitivity set to default
so you are right then do grow but not enough to even come close
to the gain you get with more b frames.

b-frames ALSO grew .5k but still are 1/3 the size of P -frames
and since I forced the b-frames Q to just 1 above p -frames
they look GREAT.

the average (all frames ) total Q was 6.1 with defaults and 3.8
with my settigns. With such a HUGE savings because of 4 to 1
b frames the new found bits really improved the encode. There
was NO low bit rate blocky areas at all now.

the Q graph was also much smoother and tighter, a true bell
curve now.


in my tests your warning about "your assumptions are pretty bad"
is falling on deaf ears because the imporvement was night and day. :D

There was just TOO much savings and i forced b frame Q to a better
quality which totally over came the stuff you mentioned.


Well you said prove otherwise :D

[Zep]

Originally posted by lordadmira


So how many B frames should I use? It depends on ur source material. The codec will try to lock on to those motion end-points encoding them as P frames to use B frames in between. The more B frames u call for, and the higher the sensitivity, the wider and more extensive the search for periodic motions. The farther into the future it will look. Generally, this is good. With experience u will just "know" the B frame potential for a particular video. The B frame "window" has to be large enough to encompass most of the motions in the video. The motion periods dictate the requisite B frame settings.

In my encodings I usually set B frames to 8/1/0, meaning max of 8 in a row and full quality. And sensitivity to 80. With this kind of a setup P frames become defacto I frames and the main load is carried by the B frames. Watching the Xvid status window, I can tell u that this does result in drastically smaller B frames of 4-6 in a row. Now I mainly encode anime so it might not be right for ur purposes. But I have gotten very good results with live material. So my advice: go for it. If it looks crappy then u have some strange material, try lowering sens and raising offset.

LA

great post too! thanks for the info.


this is the first encode I did on my test source
that had a *HUGE* improvement with just Xvid settings change.
Night and day jump in quality. Until now it was
small improvements but this was woaH!!!

My test source is BIT STARVED like crazy american
idol episode. Live and HIGH MOTION and LOTS of panning
and pull backs and fast cuts. Read my ealier post for
some details on the lastest test.

Normal is 1100 kbits at 512 x 384 and this show needs
2500kb rate to hit Q=2 on all frames.


I see you set to 80 but do you ever get over 4 to 1 with that.
You say 4 to 6 to 1 but i tried 75 and I was getting max 3.4
to 1 here. I had to set to 100 to hit 4 to 1. hmmm...



for me the bit savings far out weigh any of the warnings
sysKin gave me. Everything he said happened but not to the degree
he seemed to be warning.

I do feel the defaults are bad now. B-frames should be
default to 2 with sensitivty to 20 and force bframes to
1 over P instead of the default to make up for more
b frames. My test showed a really even encode now. No
low bit rate parts and blocks. (I tried bummping all three
overflow values to better use the bit bucket when needed
but it was only a small improvement and it also made the
detail go flat in scenes that needed it) note only
set to 2 in a row. I think more than that is only
really needed for starved bitrate stuff.

So would I do this if the encode had plenty of bitrate?
no. i would stick to 2 b-frames or less.

this is an extreme case of a very hard to compress video
that has to hit a certain low bit rate target size and still
look decent. (and now it looks rather good even)


thx

[Zep]

Originally posted by Pen-Pen
Interesting settings indeed ! I never thought about going for a lot of consecutive and hi-q b-frames... That's something I'm gonna try :) I usually go for max 2 Q4-BF...

when i first started to test this I too left bframes at Q=4
but a long run of b frames 4 to 1 or higher looks bad even with
the HUGE savings. Forcing a better Q found the balance.
Sure B frames size shot up but they are still 1/3 the size of
p frames and the end result was much better this way than
the defaults.

[lordadmira]

Originally posted by Zep
I see you set to 80 but do you ever get over 4 to 1 with that. You say 4 to 6 to 1 but i tried 75 and I was getting max 3.4 to 1 here. I had to set to 100 to hit 4 to 1. hmmm...
Well there again it depends on ur source material. It's wrong to think "I want a 4 to 1 B frame ratio". The B frame ratio is dictated by the particular kinds of motion in ur source. U have to just trust the algorithm to make the best decision. 80 seems to give about as good a result as is possible. 100 didn't make any real difference for me. Sometimes 3 B frames really is the best way to handle a motion. Sometimes it's 1, sometimes it's 8. That being said, I think there is room for improvement in the current search algorithms. I would love to see an advanced option for more sophisticated periodic motion detection. Of course this will be of the biggest advantage to anime encoding. :cool:

To understand where syskin and the defaults are coming from u have to realize that that is just one paradigm for B frame usage. The defaults are geared to "cheating" on the quantizers every few frames in order to save space. The way I use them is another paradigm. By having lots of full qual B frames I am effectively implementing keyblocks and subkeyblocks. And if u ask me that is the future of video compression technology.

The future for B frames is very bright. Right now they are limited to referencing proximal frames. Future codecs will let them reference arbitrary frames which will squeeze even more savings out of periodic motions. So where is that Xvid-2 CVS tree... :D

LA

[Zep]

Originally posted by lordadmira


To understand where syskin and the defaults are coming from u have to realize that that is just one paradigm for B frame usage. The defaults are geared to "cheating" on the quantizers every few frames in order to save space. The way I use them is another paradigm. By having lots of full qual B frames I am effectively implementing keyblocks and subkeyblocks. And if u ask me that is the future of video compression technology.


I have forced Q=2 on bframes for a long time but it was just
recently I noticed the total ratio.

I saw 4 in a row here and there so I never bothered to check the total p to b and get
THAT ratio. Then the other night I happened to glance at the stats window and noticed
the totals and I went

"WTF!!! that is not 4 to 1!"


and well here we are :D

[pelle412]

This may be a silly question, but I made a 5% compressibility test using GKnot 0.28.8 (XviD RC4) using 3 options:

1. Default B-VOP settings
Gordian Knot Compressibility at chosen size: 70%

2. BVOP settings 8/1/0 sensitivity 80
GK Compressibility: 37%

3. No B-VOPs at all
GK Compressibility: 47%

The movie used for the test is Beverly Hills Cop. Is the low compressibility result for the high B frame usage as a result of the particular type of motion in the source? Since B-frames are usually smaller than P-frames I kind of expected a better compressibility test result.

I'm a little confused...

[Asmodian]

I don't think compressibility tests are a good way to test B-frame settings as they only use 5% of the frames.
As lordadmira has been saying B-frames are useful (with ratio/offset of 1/0) when encoding periodic motion, using 5% of the frames doesn't properly represent this motion. This means B-frames will try to predict off of P-frames which have nothing to do with them to a much greater degree then normal.

I don't think compressibility tests are useful for testing most codec settings like B-frames and rate control (Though they might still apply for matrix selection and speed/quality settings).

[pelle412]

Ok I tried a compressibility test over the whole movie and got 118% so you were right. Thanks.

[lordadmira]

I have no idea how Gordian Knot does it's compressibility test so I can't really comment on those results. It could be that it's making certain assumptions that don't hold up anymore with the kinds of settings we're talking about. But yeah those tests are of dubious value if u ask me. The best compressibility tester is the human eyeball. Just watch the movie. How much motion is there? How much *noise* is there? Encoding the damn noise takes up more space than the actual images. Using appropriate filters can usually take care of that. Really, u just gotta develop the eye. :cool:

LA

[sysKin]

Not trying to spoil your fun ;) let me just comment this:
Originally posted by lordadmira
[B]To understand the power of B frames u need to realize what type of frame sequences they can exploit. There are two basic types of motion in a video, progressive motion and periodic motion. (...) This too is progressive motion and is best handled by P frames. This is because the image information for the moving object constantly requires new information, not information that has already been encoded(about the moving object).
Unfortunately I dissagree. It is the nice, linear "progressive" motion that is handeled by b-frames best. B-frames have complex motion prediction which makes such motion handled best - the most simple macroblock mode assumes that the motion is "in between" reference frames, and it is this mode that makes bframes powerful.

If this does not happen, bframes still have three more macroblock modes that are nowhere near as good, and are definitely worse than their p-frame equivalents.

Now consider periodic motion. This is exactly what it sounds like. An object is moving in one direction and then moves back where it's previously been, or nearly so. A B frame will reuse information from both the beginning (like a P frame) and end of a motion.Ehh no? Where would you want the two p-frames to be? Just remember that b-frames don't predict any information from each other, so if motion is not linear, every single b-frame will have to repeat the same motion information, again and again. In short: no, it will not work.

B-frames algorithm can still improve of course, but the next step would be to detect *linar* motion, and *prevent* b-frames if motion is not linear....

Radek

[lordadmira]

Originally posted by pelle412
Ok I tried a compressibility test over the whole movie and got 118% so you were right. Thanks. Whoa! Compression over 100%. So the output file actually takes up negative space?! I need to get me some of them files! The more u have, the more space u have!! :devil:

[lordadmira]

Are u done editing ur message? I almost did my reply before u changed it. ^_^

Yes, all motions are ultimately composed of linear segments. It wasn't my intent to suggest that multiple periods of a motion could be encoded within a single B frame sequence. Each maxima and minima, peak and trough, of the motion has to be anchored by P frames. Think of ur classic sine wave. The origin is the I frame, the peak is a P frame, the trough is a P frame, and everything in between is a B frame. Now for the maximum exploitation of B frames the Bmax and Bsens have to be large enough to let the codec *find* the critical points to encode them as the P frames. Having Bmax=2 just isn't going to let this happen. When the critical points are locked onto, the moving object effectively becomes a sprite moving independently across the background at just the cost of the motion vectors.

I think we are actually in agreement on this, we were just using different terminology. :)

LA

P.S. OK I will admit my original car example was a little flaky. That situation would be best handled by B frames. I was trying to think of a situation where B frames would actually be bad.
:scared:

[sysKin]

Originally posted by lordadmira

I think we are actually in agreement on this, we were just using different terminology. :)Hah, I was kinda suspecting that when I pressed *submit* ;)

Okay I mostly agree with you. The sine wave was not a perfect example because it is not linear - let's say that the origin is I-frame, the first p-frame must be at 60 degrees most. After that (at the peak), p-frames must be more often because it is no longer linear.

In general, don't think of a p-frame that is too soon as a waste - it isn't. There is also the problem with p-frames being too rare - there are some limits of motion estimation, and if the motion is simply too long it will not be caught. And if it will be caught, many bits will be used for it.

Unfortunately, mpeg-4's pframes were not exactly optimized for that - they are mostly optimized for low motion and low (up to mid) bitrate, with good and easy motion compensation. B-frames kinda spoil that optimizations, because p-frames become more complex.

Anyway, have fun experimenting with all that.
Radek