RC averaging period

[Psymaster]

I was browsing the divx.com forums and I came upon this (http://forums.divx.com/viewtopic.php?topic=40228&forum=6) .

What do you people say about the rc averaging period? Gej suggests putting it at half the size of the movie. But is it important? Why is the default so low? These people seem to consider it very important. Has anyone tried it?

And why half of the vid's length?:confused:

[manono]

Hi-

It seems to say how many frames ahead the encoder can look to figure out how best to average out the overflows. Doom9 has this to say about it:Rate control averaging period: How many frames the codec will look at the determine the bitrate of the frame to encode. If previous frames have used much bitrate the codec has to average that by giving upcoming frames less bitrate so it can keep the bitrate you set. The default 2000 seems a pretty reasonable value, but you can experiment with higher setting and you may find that they give better results. Setting it higher will decrease the ability of the encoder to adapt to low/fast motion scene changes and you may not get a predictable size anymore. I've been leaving it at default, but based on what I've been reading, I think setting it for half the movie may result in slightly better quality. I expect it will slow down the encoding a bit though. But maybe there are some people around here that have experimented with it and can chime in.

[Psymaster]

But maybe there are some people around here that have experimented with it and can chime in.

That's what we're waiting for!

My main concern is that setting it to half the movie's frames is way higher than 2000.

[drizztcanrender]

I tried it once...but nothing changed.The quality was tha same and speed didn't seem to change either.I don't think it will affect the quality that much,but you can still expirement.

[Psymaster]

Originally posted by drizztcanrender
I tried it once...but nothing changed.The quality was tha same and speed didn't seem to change either.I don't think it will affect the quality that much,but you can still expirement.

I tried it wih a short clip and didn't find any differences too... One question is, why should it be set @ 1/2 the movies length and not all of it?

[JohnMK]

Very good question.

^bump^

[drizztcanrender]

Well maybe because of this...

Setting it higher will decrease the ability of the encoder to adapt to low/fast motion scene changes and you may not get a predictable size anymore

What i'm thinking is that half the frames is the maximum number the encoder can have so he can see how much quality is being given to previous frames and then lowering the quality in frames after that.So if you put it in half of the frames of the encode(i.e. most of the time bigger than 2000) it will have a better average and it will also have less frames to lower ;).That's why if you increase too much it will increase the quality of the movie,theoretically,but also might get off from the predicted filesize.

Well i could be full'o s**t so, don't listen to me :)

[UGAthecat]

I've encoded several movies using the RC averaging period equal to the total number of frames and had no problems. In fact, some videos looked considerably better doing this, but I guess as with all encoding stuff, your mileage may vary.

[bkam]

I was wondering if anyone had updated information on RC Averaging. I've read in many places that it increases quality and makes file size actually more predictable because it distributes the bitrate better over the scenes.

Originally posted by drizztcanrender
That's why if you increase too much it will increase the quality of the movie,theoretically,but also might get off from the predicted filesize.

The interesting thing is I read under the sticky post about undersize (http://forum.doom9.org/showthread.php?s=&threadid=24584&pagenumber=2) that it helps him to more closely get filesize, but you could still be right if he's only talking about undersized files.

Hoping someone can clarify what this does.

Also I've heard that setting it to half will make the codec much more (not less) responsive to changes in high motion and low motion scenes, unlike what Doom9 said, but I have no experience.

[?¿öM¿?]

Hi, i am glad this thread has reemerged.
I did a few tests with the various DivX/pro settings over a few complete movies, to see how all the various things mixed with eachother.

I increased the RC averaging period to be equal to half the frames in the movies (60,000+) and on high bitrate encodes it came up with better results on the most part than the default but seemed to encode at an average of half the FPS.

On low bitrates it was easier to see the RCa at work. At 60,000+RCa on low motion scenes, the results were rather terrible but excellent on the fast motion scenes. At 4000RCa I got the best results on the fst motion movies I encoded as a good mixture between the few low and many fast motion scenes was struck, but there were edge artifacts. For mostly low motion movies I got the best results using a 1000RCa where the scenes with little panning and movement came up crisp and artifact free. However, panning for more than 5secs made ugly macro blocks appear that got worse as the panning continued to the point where it was unrecognizable after 20secs. Where a background was stationary, but objects moved around in the picture, edge artifacts appeared around the moving objects. File size also became unpredictable.

This is only the jist of what I uncovered. I did the tests at 1pass (with vDub) only as it seemed logical that the RC options would play a significantly larger part. I'm on holidays and away from my home pc for the next month or so, but if you would like, when I get back I can put together some proper notation and screen shots, maybe do some 2pass tests aswell(only did them while playing with the pro options).

[bkam]

Thanks for those results. I am currently trying Fear and Loathing in Las Vegas with an RC Averaging Period of 85000 (for 170000 frame movie), 2 pass 2cd encode. Only B-frames are enabled, lanczos to 704, pretty barebones. Bitrate 1520, so pretty high (by my standards?). I'll let you know how it comes out.

EDIT: Okay, I did this last night. It came out beautifully, with an average quantizer somewhere in the neighborhood of 2.6. This is probably the highest quality DivX I have ever encoded, but that may be more due to the fact of a 75% compressibility test, 1500+ bitrate, and lanczosresizing. No filters, just B-Frames, I'm not sure if the RC Averaging had anything to do with it. I suppose I should have tested it by encoding the entire movie at 2000, but I was in a hurry to get the VOBs off the drive.

[BoNz1]

@ drizztcanrender, you are quite right, that is exactly what rc averaging does. Say if your max min quantizers are at 2 and 12 and you have rc averaging set at 2000 it does not allow for much variation, it is like putting the codec in a straight jacket and it will not be able to compensate for it as you would like. It will have to average the quantizers between 2-12 in every 2000 frames. If it is set to half the number of frames then it will be able to average the quantizers better and get a much more accurate result. Of course, you do not have to set the rc averaging to half the frames, it is just a safe estimate. Larger values should give better quality to high motion scenes and lower values to low motion scenes. So if your movie had a lot of low motion scenes you could set the rc averaging to somewhat less than half and vice versa for a movie with a lot of high motion. This may explain why some people have different results if they set it to be much higher or much lower than 1/2.
@ ?¿öM¿? I am interested to see your results if it were a 2-pass encode. The rc averaging is needed for both passes so testing it with only one pass it may be harder to see the difference that it can make although your results definitely support what I have just said. What you have done is like a smart-1-pass encode instead of the usual 1-pass encode which is completely blind and usually gives unpredictable results.
I did The World Is Not Enough about a year ago with rc averaging set at 2000, using a bitate of about 1500, using b-frames and gmc and doing a 2-pass encode. The result was awful, it looks disguisting especially high bitrate scenes which had blocking everywhere. I was a bit of a n00b at the time so I don't think I really noticed too much. But since then, I have always set it to half and have been quite happy with that, high and low bitrate scenes look much better, and filesize predictability seems to have gotten a lot better. I would really like to encode that movie again though once with it set at 2000, then 1/2, and then maybe even try a 1/4 or even 3/4 just to see what it looks like.

[?¿öM¿?]

Hmm, I am getting really interested in this topic again and am going to do some comprehensive research.

I always use 1/2 frame count for my RCa on 2pass encoding. It gives me the best distribution of bits. I think that i'll write a prog for comparing the motion values versus bits from the divx log file and analyse.log from 2 encodes of the same movie of different RCa and let the computer do my thinking for me.

To explain:
I take 2 DivX movies, and their respective log files.
I will take the motion values from the 2 DivX log files that were created in first pass and if they are not the same(haven't checked that) I will average them out to another file(motion.txt).
Next I will compare the number of bits that is shown in analyse.log on a frame by frame basis with the corrisponding motion value for that frame. I will do this for both encodes. It will require a bit of research to decided what to say is optimal bits for motion value versus given (average)bitrate, but I will come up with something.
Then the prog outputs into a textfile the difference(+ or -) between the optimal bits and the actual bits.
The two resulting textfiles are then compared and the results are outputed into a final textfile that tells which encode was better at each frame and which is the best over all.

The parametres:
Same average bitrate given to the codec.
BiDirectional encoding disabled. Reason being, the 2 encodes will probably have I,P and B frames everywhere. I don't want to check different frame types against eachother.
If from one encode, a frame is an I-frame and from the other it is a P-frame, the frame will be dismissed.
No GMC or q-pel.
Quantizer set on 2-12.
2passes.
The actual program will take bitrate given to the codec and directories where the DivX encodes and logfiles are located. I say 2 encodes the whole way through, but I will design it so many more can be included and all compared against eachother.
As the encodes will have the same given resizing methods and no filtering and the DivX log file records are from an all intra frame 1pass, it is safe to assume things like the detail from the input(my.avs) to the codec is the same, so I shouldn't need to compensate for the texture value.

I made all this up as I went along, so I haven't thought this through yet and I will begin to do so. Can't begin it yet(when back from holidays). I would like some feedback as to possible adjustments and the feasibility of such a program. If anyone thinks this is a really silly thing to do, please tell me and why you think so, don't let me go ahead and waste my time.:D

[bkam]

I'm not a programmer so I'm not sure whether I can help you with feasibility or anything like that but I can tell you that this is a subject that also definitely interests me. I've been trying for a long time to get hard information on this subject, everyone has a "do your own tests" mentality and I agree that that is the best way to make up your own mind on a subject, but sometimes it's tough to do repeated tests especially when the idea is doing a bunch of 2-passes on different movies. With my limited processor and harddrive space it's not easy for me. Also I'm a newbie so interpreting the results of analyse.log isn't exactly easy.

Your test results would be of great interest to me, and if you are planning to release the program i will definitely make an effort to help test it and post results. I definitely support this research.

[theReal]

I'm not sure about the quality improvement, but I have experienced some serious undersizing when I set the RC a.p. to the full amount of frames. The DVD I'm talking about was the Futurama season 1 DVD set where single episodes came out with 165MB instead of the 175MB that I was aiming for.
With the RC a.p. at half of the total amount of frames I have only had minor undersizing (like 695MB instead of 700MB), but mostly it was pretty accurate.

[leadman584]

Hi

I've been playing with averaging period for a couple of weeks now, just to see if there was indeed any difference. By and large I have gotten slightly better results increasing from the default 2000 avg. Basically the default will average bit allocation over a span between 67 and 83.4 seconds depending on your framerate. I just did xxx a couple of days ago and I can assure you that to put it on 1 cd (not a real good idea, just too much high speed action) the default setting of 2000 produced incredibly bad looking high motion scenes, most vastly exceed 84 seconds. low motion scenes were ok, not great, but ok. I tried it again with averaging set much higher, 70000 I believe. Anyway the improvement was very noticeable, though low motion scenes showed signs of background pixelation. Tried one last time with latest stable build of Xvid, all stock settings. the results were by far the best of a single cd attempts. In the end I used Xvid 2cd for archive copy, and the results are stunning. It helped that I could increase the sound quality by going 2 cd.

I think by and large setting the RC averaging to a higher level for high action movies is very helpful. Perhaps others have gotten different results, but when you really push it with relatively low bitrates the difference becomes apparent. I have posted in the suggestions section at Divx that the default averaging period be set to 60000 or higher for the next release of Divx. If you have better ideas for this please post at Divx.

[yingx2]

So I guess that I am the only one who cannot see any difference introduced by this RC averaging period.

I did many tests too :mad:

[bkam]

Originally posted by yingx2
So I guess that I am the only one who cannot see any difference introduced by this RC averaging period.

I haven't done extensive tests but I can't see much of a difference either. My laptop isn't exactly state-of-the-art, though, so maybe I just don't have the hardware to tell.

[dTb]

I haven't really done any comparison tests myself but I always set the rc averaging period to 1/2 the # of frames.
No one has confirmed it for me but my opinion of setting it to half is so that you can retain utmost filesize predictability, eg. the rc averaging period ends on or very close to the end frame. By this reasoning you could set it to one third or one quarter etc.
If you were to set it at three quarters for example the video would finish part way through the averaging period, potentially messing with the filesize predictability.

[bkam]

Originally posted by dTb
I haven't really done any comparison tests myself but I always set the rc averaging period to 1/2 the # of frames.
No one has confirmed it for me but my opinion of setting it to half is so that you can retain utmost filesize predictability, eg. the rc averaging period ends on or very close to the end frame.I'd never heard it explained this way, it may well be true but I don't know. It sounds reasonable to me. I have noticed like you have that my files come out exactly perfect size this way. Out of curiosity, if you are correct about how this works, why not set the RC averaging to exactly the number of credits? Wouldn't this also do the same thing? Say, for example, a movie has high motion for the first half and low motion for the second half. If you had RC at half, even though it distributes bits well in each half, the second should get less overall... Or am I misunderstanding what the period does?

[?¿öM¿?]

The RCa period is the user defined average bitrate period. If you have a clip that is 120000 frames long and you give the codec a bitrate of 1200 and an RCa of 60000, the bitrate will average out to roughly 1200 in each half of the movie. If you give it RCa 2000, then bitrate will average out to 1200 in each 2000 frames.

This is why you tend to get lower quantizer values at the end of an RCa period, the codec under compensates for bitrate distribution and usually has an excess to get rid of at the end of the period. A file that is a bit too small is better than a bit too big. Theoretically on a 2pass encode, this shouldn't happen as the reason for a 2pass encode is to get a near perfect distribution of bits within the RCa. In practice this doesn't seem to be the case. It still seems to have lower quantizer values at the end of the RCa, but it isn't as dramatic as 1pass which means that the distribution of bits is better. The RCa when used in 2pass is for allowing the codec the biggest period for allocating bits and not forcing it to spend them in the wrong places as a low RCa will do. You get close to your desired file size using 1/2 total frames because as the RCa will only roughly come out at the average bits, it is usually less, the accumulation of the little bitrate underrun errors is less as there is only two of them. If you use 2000RCa with a 120000 clip, there will be 60 of them.

On a one pass encode, you use the RCa to force bitrate spending(or not). On a low motion movie, using a low RCa(500-1000) will produce better results as the codec spends bits faster as the end of the RCa period is so much closer. Often the RCa period will only have 2-4 I frames and the rest are P&B(in lowmotion). This means that as there are fewer bits needing to be spent, it will lower the quantizer to get a better image to use the bits instead of conserving them. However, the moment fastmotion comes in, the codec freaks out because bits are being spent rapidly and it has little time to average them out. It then increases the quantizer value and begins to throw P&B frames where it shouldn't in order to conserve bits and the result is incredibly ugly macroblocking.

(FastMotion definition: The more pixels that change colour from frame to frame and the bigger the colour change, the faster the motion) In movies with lots of fast motion and an RCa period of 4000-6000, the best results will be achieved. The codec will hold off on increasing the quantizer early in the RCaPeriod during fast motion as it sees that it has alot of frames left before it needs to achieve its average bitrate, so it will wait and see what happens. However, when it comes to a low motion area where it could easily spend bits and throw low the quantizers around willy nilly, it won't because it doesn't want to spend bits early in the RCa Period, resulting in poor quality low motion scenes.

Of course when you are using stupidly high bitrates, the RCa Period tends to become moot as the codec has heaps of bits to plaster all over the place:D . It's effects will become less and less, the higher the bitrate.

I will at some date do extensive testing of the Rate Control Averaging Period on 2pass, right after I finish shooting holes in WM9series;) .

[bkam]

@?¿öM¿?:
Thank you for your thorough and informative response.

However, I still have a few questions. I now understand the nature of the averaging period much better. But I do not understand, if this is the case, why you would not just want the 1200 bitrate you mentioned to be distributed among all 120000 frames at once. In other words, why not allow the codec to look at the whole movie and judge the distribution of bitrate over the whole thing?

I understand what you say with low motion, that setting it low will force it to use bits towards the end. Is this to say that the codec will reduce quality in low motion scenes disproportionately? What I mean is, it should be fine if you set a large RCa and then it sees a low motion scene and proportionately reduces the bitrate (because low motion requires less bits than high motion, so in theory it's the same quality). But I think what you are saying is that the codec will give even fewer than is necessary and decrease the quality in these scenes?

It seems to me, if the RCa has no upper limit and the codec is "smart" then why not set RCa to half the movie, or all of it? I can only infer that the codec is smart, but by limiting the size it has to average, you can more accurately assign bits. I would appreciate confirmation (or negation) of this inference.

[cordraconis]

I'm in the middle of an exam period, so this thread doesn't completely "penetrate" my tired brains :D But here is a good suggestion for the persons who do have time for this:
- If I'm correct, then putting the RCa to the maximum ammount of frames, results in undersized files.
- If you run a compressibility test, and it's too high, you also risk undersized files.
- What if you use a DivxEnc compressibility test, and use the percentage (and therefore a number that represents the ammount of HM-scenes) as an Empirical Law to find the optimal amount of frames? (lots of testing required ofcourse, but if you can first agree on a rule op thumb ... I suggest you take the percentage above 75%, double it, and then use that percentage to reduce the percentage-of-frames (which is standard on 50%)

This means: Comp Check is 100% --> +25% Diff. compared to 75% --> 50% Standard minus (2*25% Diff.) is 0% RCa --> One frame RCa (Makes sense since compressibility is 100% !!!)

Or Comp check is 50% --> -25% Diff. compared to 75% --> 50% Std. plus (2*25% Diff.) is 100% RCa --> The codec should use *All* the frames to distribute the bits as efficiëntly as possible. (Something like a 1-pass encode, with the quality parameter according to the desired bitrate/filesize).

I'm a Chemist and exact mathematics are *not* :( my style, but I love "real" laws :D that are not exact, but good enough for reality. This is just a suggestion from me, if anybody has better "values" for the above (like for example bias everything on 60% in stead of 75%) then go ahead.

[bkam]

@cordraconis:
Interesting idea. I thought I'd simplify the expression so it's easier for anybody who wants to test it..

Let x be the compressibility test value.
So the the expression to get the % above 75 is x - 75.
You say to double it, so 2(x - 75)
You say to reduce the standard 505 by this.
Therefore 50 - 2(x - 75).
This simplifies to 50 - 2x + 150 = 200 - 2x.

So you can use 200-2x, where x is your compressibility test. To use your examples, a test of 100% = 200 - 2(100) = 0 and a test of 50% = 200 - 2(50) = 100%. That makes the "standard" 50% RCa period happen at 75%.

I'm not sure if this works, particularly because certain people seem to think that over half frames for RCa period may not be smart (dTb for example). I'm not sure which is true but

[SpEeDaMiGo]

@bkam:
i won't comment on all your maths, let me just tell you this: if you want (nearly) perfect bitrate distribution then go for divx3 sbc (or xvid as well, i think). i consider divx5 2-pass mode as kinda pseudo, cause, as you mentioned, it doesn't really analyse the movie as a whole. with divx3 sbc you perform a first pass which is bitrate-independent, cause it just analyses all the frames without already allocating bits to it (divx5 somehow does), which is then done "before" the second pass and which can be tweaked almost infinitely and which provides a real control over the bitrate allocation, whereas divx5 is a mix of 1-pass and divx3 sbc. i'm no expert on this, but that's what i understand of the whole matter. well, to sum up: you won't get perfect bitrate allocation with divx5 and even tweaking the averaging period value will always have some drawbacks concerning either size predictability, slow or fast motion scenes.
hope, this gives you some rough idea.

[ohliuv]

If I remember correct, someone mentioned in the forum that 50% RC significantly improves the overall result of hard to compress movies.
Also remember that some time ago I experimented with SPR (full encode), and the result looked better with 50% RC.

[UGAthecat]

I thought the main reason people were using 50% for the RC avg period was because when splitting one avi for 2cd rip, if you use 100% of the frames you can't always split dead center in the movie.
so, when you use 50% of the frames as your RC avg period, the first half and second half of the movie should be the exact same size, allowing you to cut the movie at whatever frame # is in the middle of the movie.
(IE: 1cd rip, use 100% for RCa, 2cds use 50%, 3 cds try 33.3...%)
also, if you calculate the RC avg period this way, you use one formula to determine both the RCa, and the frame numbers you split your file on if aiming for more than one disc.

[bkam]

Does that mean that you automatically have a keyframe every RCa period number of frames?

[drizztcanrender]

Does that mean that you automatically have a keyframe every RCa period number of frames?

That sounds possible.It might put keyframes at the beggining and at the end to create boundaries which will help it analyze.But i haven't a clue if it is really true.


originally posted by SpEeDaMiGo

i won't comment on all your maths, let me just tell you this: if you want (nearly) perfect bitrate distribution then go for divx3 sbc (or xvid as well, i think). i consider divx5 2-pass mode as kinda pseudo, cause, as you mentioned, it doesn't really analyse the movie as a whole. with divx3 sbc you perform a first pass which is bitrate-independent, cause it just analyses all the frames without already allocating bits to it (divx5 somehow does), which is then done "before" the second pass and which can be tweaked almost infinitely and which provides a real control over the bitrate allocation, whereas divx5 is a mix of 1-pass and divx3 sbc. i'm no expert on this, but that's what i understand of the whole matter. well, to sum up: you won't get perfect bitrate allocation with divx5 and even tweaking the averaging period value will always have some drawbacks concerning either size predictability, slow or fast motion scenes.

I will have to disagree,it's not pseudo at all.Maybe you are talking about divx4(?).The first pass of divx5 is done with quant 2 and it is bitrate independent.It's job is to analyse every frame of the movie and then store the quant values of every frame in the .log file and the motion estimation values in the .bin file.Then in the second pass the codec just uses all those informations and to encode the movie.
So i would say the divx5's 2-pass method is better than sbc.

I have a suggestion for you guys trying to test the rc averaging period.You can try this,although i'm not sure how much it will help. Try to do some 2-pass encodes of the same clip,with the same values for the codec options and changing every time only the rc avraging period.Then make a program that takes the quant values from a .log file and create graphs to see if there is any difference.You should be able to see clearer quant distribution in a graph.Just a thought,i may be terribly wrong but i am tired so... :)

See ya...

[jonny]

The first pass of divx5 is done with quant 2 and it is bitrate independent


Sorry, it's not correct, divx5 first pass is done using the bitrate you specify (thats why you can't change too much to bitrate from the first to the second pass)

[theReal]

Originally posted by jonny
Sorry, it's not correct, divx5 first pass is done using the bitrate you specify (thats why you can't change too much to bitrate from the first to the second pass) Yep. I guess drizztcanrender you were thinking about the Gknot compress test. Gknot sets quant max=2 and quant min=2 for the compress test first pass.

There have been attempts of changing the Divx 4/5 1st pass log with a tool called Morphix. The results were not really worthwile, as far as I can remember because nobody except the Divx people know exactly how the log is used for determining the second pass quantizers (and they won't tell, obviously...)