Calculation of compressibility check

[zap_path]

I've been encoding some movies into Divx that have reported very large bits per pixel (as high as 1.7 in one case). I think part of the reason is that the video I'm encoding was hard telecined, which if I understand the process right, would have dramatically reduced the compressed video quality and seems to have resulted in noise in the picture making my recompression difficult. I guess the correct way to deal with this would be to apply some kind source pre-processing, but this doesn't seem to have much effect on the number returned by the compressibility check. Even when the pre-processing is set to "strong" the compressibilty check is only reduced from 1.075 to 0.907 bits/pixel*frame. Encoding the video, at a reasonable size results in fairly poor results. I'd like to do better.

In order to figure out what to do, I would like to know how the compressibilty check is really computed. To find out, I checked out the log file produced by the compressibility check. I summed the total_bits column, and divided by nframes times the pixels per frame. The result was 1.001 whereas GordianKnot reports 1.075 bits/pixel*frame. I thought the difference might be accounted for by the fact that the compressibility check is run with a quantizer of 2. Given the explanation in the Divx manual, this sounds like it would remove one bit from every record. However, the Divx setup dialog reports a percent savings for different quantizers that doesn't match the savings predicted by the explanation of quantizers given in the manual. That explanation leads to a size as a fraction of the full size of:

1-log2(Q)/N

N: original number of bits per record
Q: quantizer

This function does not match the numbers reported in the Divx dialog for any choice of record length. That function appears to be closer to linear than logrithmic. So, I can't account for the difference between what I calculate and what appears in GordianKnot. What accounts for the discrepency?

Also, any advice about what to do about very large compressibility check results would be appreciated.

Thanks,

-zap_path

[manono]

Hi-

I think you're making this harder than it really is. Forget that bits per pixel stuff. What you're looking for after running the compress test is the resulting percentage.

If your material really is hard telecined (and you do understand what that means, right?), then you encode with IVTC on after having made your Movie.d2v with no Force FILM (29.97fps). In such cases where you apply IVTC, you then have to change the framerate in the lower left hand corner of the main GKnot screen to 23.976 to get an accurate percentage figure.

Also, any advice about what to do about very large compressibility check results would be appreciated.

You mean like over 100%? Then you do the usual things; raise the resolution, give it better quality audio, leave off the B-Frames, lower the file size, don't use smoothing/denoising filters, etc.

[zap_path]

Thanks for your comments.

I guess from a practical point of view, you could just look at the percentage and that would be enough, but as an engineer, I like to how things work underneath the surface. In fact in my case, the percentage number isn't all that meaningful because I'm always going to encode a 45 minute video onto half a CD, and watch it on full screen. So the dimensions of the encoding aren't critical.

The result of the compressibility check is not a percentage but is rather a bits/(pixel*frame) number as reported to the right of the percentage in GKnot. Since I used Telecide and Decimate and used the final number of frames encoded (not the original) to calculate bits/(frame*pixel), the change in the frame rate shouldn't affect my calculation. I'm guessing the discrepency must have to do with my misunderstanding of how the quantizer works or something to do with GKnot clip selection mechanism.

According to the Divx manual, quantizers take some integer used to represent some image information, divide it by the quantizer, Q, round to the nearest integer, and multiply to restore an approximation to the original value. So, if Q=2, this process would eliminate half the possible values, and reduced the amount of bits needed for that value by one. Q=4 would remove two bits, etc. There are two problems with this scheme. First what does it mean if Q=3? How do we eliminate a fractional bit? Second, this interpretation does not correspond with the size reduction percentage reported in the Divx encoder setting dialog.

My other guess is that the problem has something to do with GKnot's clip selection scheme. For some reason the compressibility check requires a selection of clips all 14 frames long. I don't see why there should be such a requirement, but maybe the explanation can account for the discrepency. Also, it seems to me that with such short clips, Divx will need a lot of I-frames, and that compression results will be worse than you'd exect when actually encoding.

Yes I know what hard telecine is. The video is reported as 0% film by DGIndex, and yet, when viewed without processing, is obviously interlaced. This is a completely insane way to encode a DVD as it's going to include lots of duplicate material and frames that look much more complex than necessary and are therefor are hard to compress. A curse upon the DVD manufacturer.

I probably wasn't clear before when I asked what to do about "large compressibility check results". I didn't mean the percentage, but rather the bits/(pixel*frame) number. For other videos, this number is around 0.3 or maybe 0.5, but never before as high as 1.7. With these high values, getting reasonable compression quality is nearly impossible. I've gone through the Divx log files and found that problem seems to be excesive numbers for texture_complexity. I think this is indicitive of a noisy source. The video looks decent when played directly from the DVD, but the noise seems to destroy compressibilty. It's annoying that barely perceptible noise should so dramatically reduce compressibility.

Thanks,

-zap_path

[manono]

Hi-

Geez, I thought I wrote long posts. :)

the percentage number isn't all that meaningful because I'm always going to encode a 45 minute video onto half a CD

Assuming all other variables are constant (resolution, B-Frame settings, etc.), then you're going to get wild swings in quality (meaning the final average quant). You can even out the quality a bit with judicious use of filters. But the compress test is still one of the best things to come along as far as helping to determine the final quality before doing the encoding. Although it's common practice, I always thought making episodes all the same size was pretty dumb. Even with 2 episodes per CD, the compress test results may tell you to make one for 325 MB, and the other for 375 MB, in order to give them both roughly similar quality. Don't believe me? Then run some of the ones you've done already through DRFAnalyzer (http://www.geocities.com/analyzerDRF/) and compare the quant averages between 2 episodes on the same CD.

The result of the compressibility check is not a percentage but is rather a bits/(pixel*frame) number

That's right, and it's expressed as a percentage of the original bits/(pixel*frame).

You've got the quantizer stuff wrong. Originally, before B-Frames, quant 2 was expressed as 200/2=100%, the best possible. Quant 3 was 200/3=67%, quant 4=200/4=50% of the original information. Before B-Frames threw a monkey wrench into the equation. Now a quant 3 is roughly 60% of the original info. But it varies, at least partly depending on your B-Frame settings.

I don't see why there should be such a requirement...

There isn't any such requirement. That's for a default compress test of 5% of the frames, giving a good tradeoff between time to get results, and quality of results. If you want to do a 10% test, with maybe (and maybe not) greater accuracy, then it takes 28 of every 280 frames. Look at the bottom of the compress test .avs. You'll see (for the default 5% test) SelectrangeEvery(280,14). It takes 14 of every 280 frames throughout the entire movie, and then when done tosses out the first and last frames (the first frame for the reason you deduced-artificially set I-Frames), leaving 12 of every 280 frames on which to base the results. You can fill in any set of numbers you wish. The people that devised this test knew what they were doing. Would you like to see some compression test results using different percentages of the original movie? Check here (http://jonny.leffe.dnsalias.com/divx5enc/cttests.htm) and here (http://jonny.leffe.dnsalias.com/enc/sp/divx.htm).

This is a completely insane way to encode a DVD as it's going to include lots of duplicate material

Yep, you know what hard telecine is. Part of the problem is obsolete encoders in use, particularly overseas, which, evidently, are incapable of performing on-the-fly IVTC, and thus can't encode 24fps progressive frames when fed an already telecined source. Most Hollywood movies are at or close to 100% FILM these days. However, many DVD extras, even those that could have been encoded as Progressive, are still hard telecined and encoded as Interlaced. I have no idea why.

For other videos, this number is around 0.3 or maybe 0.5, but never before as high as 1.7.

I agree that's a ridiculously high number. Are you encoding at very high resolutions or something? The only time I've experienced numbers even close to that (although at 512x384) is with silent films, which sometimes put to shame almost anything going today, as far as lack of compressibility goes. But grain/noise is very hard for MPEG-4 to compress and is, in my opinion, one reason why it still isn't as good as MPEG-2. Give XviD or DivX a good clean source, and they're great. Give them a real grainy source (Saving Private Ryan, for example), and they fail miserably. But to aid in compressing such sources, there are a number of filters you can use, such as RemoveGrain, Convolution3D, TemporalCleaner, or FluxSmooth, to name a few. Everyone has his/her own favorites.