Hardcoding Alt-CC and CC: which settings to use

[Acaila]

XviD has been out for quite a while now, and everyone has had plenty of time to test out all the interesting features/options. Two of the most challenging and difficult to understand settings are the Traditional and Alternative Curve Compression.
I think it would be best if we could come to a general consensus about which settings would be best for each type of movie, so those settings could be hard-coded into the codec. That way less experienced (new) users could select from a profile list the type of movie they're encoding and still get the benefit of great settings. Ofcourse there is no best way for each type, but I think we could at least get to a few settings that are considered to work generally well.

Hence I've tested some traditional CC settings for the movie The Matrix (R2 PAL), which I believe has a nice mix of motion and brightness to see the effects of different settings.

These CC settings were tested with the following base parameters:
Target Filesize: 720.000kb (with 95MB audio gives a total of 800MB)
Resolution: 640x272
Search Precision: 6
Quantizer Type: MPEG (both passes, modulated would give unpredicatable results)
Max I-Frame: 250
Min I-Frame: 5
Lumi Masking: Enabled
I-Frame Range: 2-3
P-Frame Range: 2-10
Credits Range: 186188-196155 (encoded with quant 31)
Resize Filter: SimpleResize
IDCT Type: 32 bits MMX SSE

Results:

I-Frame Payback Payback
Boost % High % Low % Delay Type 2 3 4 5 6 7 8 Average Quant
10 0 0 500 Bias 20953 143327 21380 426 102 0 0 3,01
10 10 500 Bias 25700 137029 23164 293 2 0 0 2,99
10 20 500 Bias 29691 129910 26143 444 0 0 0 2,99
10 30 500 Bias 33546 122548 29404 687 3 0 0 2,99
10 40 500 Bias 37262 115237 32639 1047 3 0 0 2,99
10 50 500 Bias 40860 108032 35669 1624 3 0 0 2,99
10 10 0 500 Bias 23354 140348 21743 636 95 12 0 3,00
10 10 500 Bias 28266 133655 23648 615 3 1 0 2,98
10 20 500 Bias 32253 126377 26651 900 7 0 0 2,98
10 30 500 Bias 36124 119112 29639 1305 8 0 0 2,98
10 40 500 Bias 39857 111849 32628 1840 13 1 0 2,98
10 50 500 Bias 43439 104770 35364 2584 30 1 0 2,98
10 20 0 500 Bias 25997 136826 22150 1122 88 5 0 2,99
10 10 500 Bias 31069 129817 23971 1314 16 1 0 2,98
10 20 500 Bias 35066 122678 26720 1698 25 1 0 2,97
10 30 500 Bias 38983 115376 29404 2370 51 4 0 2,97
10 40 500 Bias 42596 108366 32071 3069 82 4 0 2,98
10 50 500 Bias 46219 101503 34415 3921 125 4 1 2,98



As you can see, the difference between average quants isn't very large. Maybe this movie isn't such a good example after all, because I've seen much larger differences with other movies.

I would encode some more settings, but my HD is currently full and I'd have to delete some of the above samples to free space. If any of you want to see some screenshots, or short clips let me know before I delete them.

Now for which setting is best:
I tried comparing frames/scenes between the various outputs, but after scanning through 18 videos I can no longer see a difference :)
So I'm resorting to a more mathematical comparison:
- Give each setting a rank based on what the average quant is (lowest average quant gets rank 1, second lowest gets rank 2, etc.).
- Give each setting a rank based on the percentage of frames that fall in the quant range of 2-4, since I would cut off anything higher than 4 on the final movie (highest gets rank 1, etc.)
- The one with the lowest total rank has the best of both worlds, and would be the winner.
Btw, I used the full floating point values to determine rank, not the rounded ones you see in the table.

Based on the above the following setting comes out as being the best:
I-Frame Boost (%): 10
High (%): 10
Low (%): 10
Followed by:
I-Frame Boost (%): 10
High (%): 20
Low (%): 20
As second best.

I can say that the other two movies I tested (X-Men and The Mummy, although not as extensively as The Matrix) both resulted in 10/25/25 as being the best (I didn't test 10/20/20 on those).
So overall I can say that 10/20/20 has a fair shot at being a very good setting for the type of movie that The Matrix, The Mummy and X-Men fall into (dark movies with a good mix of action and low motion).

Now if more people would compare some settings on special movie types we can discuss what options to put into a profile list or something. I'm sure there are enough XviD experts out there already who can help with this ;-).


Regards,
Acaila

Ps.
I could have put this into the Alt-CC thread, but that one is getting too large already and I would like to discuss both Alt. and normal CC results here.

[OUTPinged_]

Your way of judging resulting quality is incorrect.

Average quantizer is not supposed to change at all.

Curve compression settings' tweaking is a quest of distributing bits in the most efficient way between large and small frames.

[Acaila]

Well then they both must be broken, because every test I've run since the beginning of XviD showed a change in average quantizer.

EDIT:
Another thing, Lumi Masking is also supposed to redistribute bits, but instead it increases compression. By quite a fair amount too I must add.

But if anyone has any better ideas of judging quality (except from watching everything, because that is too subjective to me), I'd be very interested in hearing it.

[Teegedeck]

Originally posted by Acaila

EDIT:
Another thing, Lumi Masking is also supposed to redistribute bits, but instead it increases compression. By quite a fair amount too I must add.

Thanks for taking the time, I'm still choking on the results and test-setup (50-10%-distances!? whoops, I've overlooked that you meant values for traditional curve-compression...:)). Lumi-masking is meant to increase compression, not to re-distribute bits - because it uses higher compression on certain areas of the picture. So if you use it in first pass already (I'd say, it's a good idea) the stats show a lower bitrate-usage/better compressibility of the movie than without it and consequently in 2nd pass, frames get encoded at a lower base-quantizer than without lumi-masking in both passes. It also does some redistribution that way - dark scenes get an overall stronger compression.

I'm sorry if I keep on nagging about your settings but they really deviate a lot from what most people use, so your results might be not typical for what most of us get. For example I-frames shouldn't be more restricted than P-frames; we've agreed that it is much more effective to just let the I-frame-boost increase their quality. A tight restriction like your maximum I-frame tends to utterly mess up the curve (this should explain why you get different averages quants where you shouldn't, too).

Yeah, it's tough to judge quality - in the end it's up to subjective taste, but I'd say it's a good idea to call one setting a good setting if it doesn't produce a lot of frames encoded at > quant8, but has a distribution with some bias towards the number of frames encoded at average quantizer (or even lower). I mean that's kind of like the idea behind curve-compression isn't it - compressing a FEW frames stronger in order to have some MORE frames encoded at a lower quantizer. And here it begins to get subjective: which settings produce the best gain/balance of increasing the number of low-quantizer-frames by compressing big frames strongerwithout that compression getting annoyingly visible ? Apart from that I have no new ideas about judging quality.

[Acaila]

I set my I-frame range to 2-3 because I've seen a long time ago that it produces better results than keeping them at 2-31. It's a proven improvement for me, and I've used it for quite a while now. Whether or not it screws Curve Compression, I really don't care if it means getting better results without having to use CC in the first place. Would you really like it if the codec decides to use quant=5 for a keyframe, which in turn the following P-frames get referenced by? I'm willing to negotiate a range of 2-4 though ;-).
Setting P-frame range to 2-31 is absolutely not nessecary, since all encodes never use quants over 10 anyway. If they are you're doing something very wrong, and not even Alt-CC can help you then :).

which settings produce the best gain/balance of increasing the number of low-quantizer-frames by compressing big frames stronger without that compression getting annoyingly visible?
The only real thing you have to weigh for yourself by using CC/Alt-CC is whether you justify a decrease of quant=3 for a resulting increase in both quant=2 and quant=4, or rather an increase in quant=3 for a decrease in quant=2 and quant=4 (90% of the movies can be restricted to quants of 2-4, in PAL-world anyway :)). And you can't tell me you can see annoyingly visible compression artifacts appearing at quant=4 no matter what settings you use, so the effect of CC/Alt-CC will be very minor IMO.

But I'm open to suggestions for better test-settings if that will make it more convincing to people....we will probably have to come to an agreement about test settings anyway so more people can jump in and we can compare everything.

[Dali Lama]

As far as I-frame and P-frame range goes, Gej has said that changing those in Divx5 would mess up the internal calibrations of the codec. I doubt Xvid has anything yet, but I remember -h saying that the codec is "BEST" at determining the I-frame and P-frame values. So leaving both at 2-31 and using 20% I-frame boost was best.

-h even mentioned that sometimes an I-frame can be smaller than a P-frame and vice versa. So maybe leaving an wider range allows for these variations.

Although I understand Acaila's point and understand that high quant values are not going to be used in "High Quality" encodes like 2-CD, etc. However, Teegedeck may be saying that if there are to be "hard coded" settings then they should be set so that under various people's needs of filesize/qualty that it should be tuned.

To me, something similar to Divx 5's internal hard coded settings should be achieved, which btw are set to a quant range of 2-12. Maybe that's a compromise between 2-3 and 2-31? (note: Acaila used 2-10 for his P-frame range, which is similar to the Divx 5 2-12)

Take Care,

Dali

[Teegedeck]

Well, we shouldn't forget that an I-frame looks better than a P-frame even if both have the same quantizer - because they use different quantization-matrices. Yes,restricting ranges doesn't seem viable for universal presets because these should be independent of bitrate (having a 1-CD-encoding with I-frames restricted to 3 shouldn't be so smart). Anyway, I thought we were discussing alt.CC/CC-settings and ranges.

Greetings everyone!

[Acaila]

As far as I-frame and P-frame range goes, Gej has said that changing those in Divx5 would mess up the internal calibrations of the codec.
In DivX5 you don't even have the option to set a custom I-frame range. It's hardcoded at quant=2. So I am still giving XviD more relative freedom compared to DivX5 :D.
Gej said that because if people started messing with the quant range and got ugly results they wouldn't come running to him to fix it. The truth is, a range from 2-12 or 2-31 is waaaay more than it really needs. With DivX5 it's very hard to actually visualize what quants it uses for each frame, but for XviD we have DebugView which can help a LOT. So for DivX5 I would indeed advise to use the default, but with XviD I use the range it really needs, which is only 2-4 or 2-5 for P-frames depending on the length/motion of the movie. The range of 2-10 is just to see how far I can restrict it safely without messing up the movie.

I doubt Xvid has anything yet, but I remember -h saying that the codec is "BEST" at determining the I-frame and P-frame values.
If you believe the codec knows best then there is no way you can justify using Alt-CC/CC, because after all the codec knows what it's doing... The same goes for setting quant ranges. If you trust the codec not to use too high quants and mess up your movie, then by all means don't change them. However I hate compression artifacts, so I don't want XviD to decide to encode some I-frame with a high quantizer thereby messing up the entire following sequence of P-frames.

Although I understand Acaila's point and understand that high quant values are not going to be used in "High Quality" encodes like 2-CD, etc.
Who's talking about 2 CD's here? I'm talking about high quality 1 CD encodes.


Ok, maybe I moved a little too fast by using my own settings to base my tests on. It seems this entire discussion is now about things that are actually secondary to what this thread was supposed to be about. I'll repeat my tests with an I-frame and P-frame range 2-31 (happy now! :)), I'll report back when they're done. Some results from other people would be nice too....:)

Anyway, I thought we were discussing alt.CC/CC-settings and ranges.
Thank you Teegedeck :)

[OUTPinged_]

Who's talking about 2 CD's here? I'm talking about high quality 1 CD encodes.

90% of the movies can be restricted to quants of 2-4, in PAL-world anyway



These two statements above are interesting

Also, why do you think pal movies are different from ntsc in terms of compression?

Ie, I have a pal movie which produces 2.5gb first pass at 640x (the picture quality is very high/sharp, movie length is 96min). Now what settings can you recommend from your experience?

[Acaila]

These two statements above are interesting
Also, why do you think pal movies are different from ntsc in terms of compression?
I meant that I've only done PAL sources, so I don't know what quants NTSC movies would need and how compressibility changes with the IVTC'ing and deinterlacing and stuff.
But for me those statements say how things work for me.

Ie, I have a pal movie which produces 2.5gb first pass at 640x (the picture quality is very high/sharp, movie length is 96min). Now what settings can you recommend from your experience?
I don't know if you're mocking me, but I'll give you the benefit of doubt... Too many variables remain to give an exact answer (e.g. live action or anime), but I could give you a few educated guesses:
- reduce resolution to 608, maybe even 576 ( I do hope it's 16:9 :))
- use a TemporalSmoother(1)
- Audio quality: Vorbis -q2.00 or -q2.50 if the size difference isn't too great
- use Lumi Masking to increase compressibility even more
- using the above you should get to about 45-50% 1st pass size, with which a P-range of 2-5 should be enough for excellent quality.
And all that is without Alt-CC/CC, which I can't even give you an educated guess about without knowing what the movie looks like.

But this is just an extreme example. Especially for a 96-min movie.

[Teegedeck]

With your preference for constant, low quantizers, I guess you could like alt.CC low aggression. ;)

[OUTPinged_]

the topic is about CC settings.

- using the above you should get to about 45-50% 1st pass size, with which a P-range of 2-5 should be enough for excellent quality.


As quantizer 5 frames are typically 40% of the size of quantizer 2 frames, the movie will end with 4-4.5 average quantizer. Do you still use caps of 2-5 for those cases?

The 1pass of the movie had a temporal filtering added already btw.

I don't know if you're mocking me

You never know :-)

Too many variables remain to give an exact answer

Ok, to narrow this down, these values will be left constant:

1pass size
resolution (672x, 1.65AR)
resize (0.65 bicubic)
audio(227k VBR mp3)
mpeg quant.
no lumi.
credits capped to quantizer8.
1400mb to spare.
2pass ratio is about


Atm I am doing a second pass with the next settings (alt.cc)

medium/100h/25l/50mrq/2-8quant(both I and P).

The movie is 'Knockin' on the Heaven's Door'. Has a couple of action scenes but isnt hi-mo oriented. A source is of very high quality, so i want to get most of it.


Post here some cc settings you think would work best. I will post my results here, too.

[Acaila]

As quantizer 5 frames are typically 40% of the size of quantizer 2 frames, the movie will end with 4-4.5 average quantizer. Do you still use caps of 2-5 for those cases?
No, I would go for 2 CD's if I got an average quant of 4-4.5. And I would never restrict the range without first seeing what the codec does at the range 2-10.

1pass size
resolution (672x, 1.65AR)
resize (0.65 bicubic)
audio(227k VBR mp3)
mpeg quant.
no lumi.
credits capped to quantizer8.
1400mb to spare.
2pass ratio is about
-I personally believe anything over bicubic (0,0.5) looks too sharp with too much ringing...
-I bet you have super ears if you can hear the difference between 192 and 227 kbps MP3. I know I sure can't (could be my speaker setup too :)). Besides, there are codecs out there that perform much better than MP3 at such a high bitrate. You'd have to ask one of the audio freaks for that.
-In your case I would highly recommend using Lumi Masking, unless you're going to watch it on TV instead of your monitor. With such an uncompressible source you can use all the help you can get.

Post here some cc settings you think would work best.
I'd probably try the following:
I-frame Boost/High %/Low %/Payback Delay
10/0/10/500
10/10/25/500


EDIT:
I've redone a couple of the tests I've mentioned in my starting post, and I still see my average quantizer decreasing with the same settings. Actually there's not much difference in distribution between the new ones and the old ones. This is with both the I-frame and P-frame range set to 2-31.

[OUTPinged_]

10/10/25/500

Any ideas why do you use pbdelay that high?




-I bet you have super ears if you can hear the difference between 192 and 227 kbps MP3. I know I sure can't (could be my speaker setup too ). Besides, there are codecs out there that perform much better than MP3 at such a high bitrate. You'd have to ask one of the audio freaks for that.

Why "between 192 and 227"? I did run a --alt-preset standard pattern, and people who i am doing rips for dislike ogg audio.

ogg audio-->ogg container-->ogm extension-->me blamed for "wtf are you doing, we want avi rips"


It yelded 227k bitrate. Probably only aps and r3mix presets were worth a note ripping-wise. As for ogg, atm it sounds to me about same as mp3 that is higher for about 20kbit. Ogg devels are still implementing that 5.1 chan compression....


P.S. Second pass with my settings is over and here are quant. distribution results:

[548] Q:2:23787
[548] Q:3:65591
[548] Q:4:25545
[548] Q:5:6283
[548] Q:6:1195
[548] Q:7:106
[548] Q:8:18

Quantizers 7&8 are mostly in "rain" scene and are transparent.

It looks like quantizer range could be capped to 2-6 in taht case, without any quality changes. Still, xvid is looking good at quantizer 7&8(no blocking) on high motion scenes.

[Acaila]

Any ideas why do you use pbdelay that high?
Not really, the default (250) seems a bit too low for my taste. But there is no difference that I can tell so you might as well use 250.
Why "between 192 and 227"? I did run a --alt-preset standard pattern, and people who i am doing rips for dislike ogg audio.
Fair enough. For my ears MP3 caps at 192kbps, so using 227 is overkill IMO and you could clearly use the space for this movie's video stream, but if there is an audible improvement for you then it might be worthwhile I don't know.
It looks like quantizer range could be capped to 2-6 in taht case, without any quality changes. Still, xvid is looking good at quantizer 7&8(no blocking) on high motion scenes.
You could easily cap it to 5 instead of 6, you'll only cut off 1200 more frames, which is only 0.8% of the entire movie. The maximum I would cut off lies between 4000-5000 depending on the movie, I basically restrict it to the range 95% of my frames fall into.

Now if you're willing to drop res to 640 and enable lumi masking, I'm sure you could cap it to 2-4 safely and end up with an average under 3 instead of the 3.15 you have now. I'm not saying you should, but just to give you an idea.

I'm very interested in hearing what a few different Alt-CC/CC settings would change for this movie.

[soujir0u]

You'll be surprised by how much better an --alt-preset standard (VBR) mp3 averaging about 224kbps sounds compared to say an --alt-preset 192 (ABR) mp3. As for Ogg vs mp3, it's generally been agreed that Ogg sounds better than mp3 at similar bitrates (e.g. 128kbps Ogg vs 128kbps mp3). I don't know how much better though.

I guess for movies 160kbps seems like a reasonable limit for the audio (use Ogg -q5.00 or Lame --alt-preset 160). Music oriented stuff (music videos, concerts, etc) I use --alt-preset standard or Ogg -q6.00 to 7.00. Also, mp3 presets were tuned for 44Khz, so you might want to downsample the audio. Should sound the same as 48Khz if you use SSRC, unless you got bat-like ears.

Funny how this post seems like it was meant for the audio encoding forum. :p

Anyway, I noticed that my encodes always get I-frames either 2 or 3, even though the range is 2 - 31. So I guess there's not really much point in limiting the range.

[soujir0u]

Oh, and do you guys use Hinted MV?

[Acaila]

No, when you use Motion Vector hints your average quantizer will increase. I don't like that.

Besides, speed is not an issue for me.

[OUTPinged_]

@Acaila

really, the default (250) seems a bit too low for my taste. But there is no difference that I can tell so you might as well use 250.

BR_state(current status)=br_state+(deviation / pbdelay).

There is difference as long as deviation is growing.

For my ears MP3 caps at 192kbps
That may be true only for CBR streams. VBR streams are different. If not using high-end equipment, --aps mp3 equals to 320kbit cbr mp3 stream. And bitrate does vary from movie to movie, too. Atm the range for a movie's --aps was 127-241kbit.


For encodes where audio source is of bad quality, ogg -q0 is ok for me :-)

@soujirou (mm, i seen that nick somewhere, dc maybe?)

hinted mv can only increase speed for a bit. As i dont care about filesize precision, i can sometimes change res/filters and leave 1pass file from old try. So hinted ME is off.

[Teegedeck]

MV-hints decrease compressiblity _very_ slightly but have some nice effects like diminishing 'moving pixels' in static areas. I'd use mv-hints.

[soujir0u]

Originally posted by Acaila
No, when you use Motion Vector hints your average quantizer will increase. I don't like that.

Besides, speed is not an issue for me.

It figures... look at your system! I'm stuck with a mediocre Athlon 1Ghz. :(

[ckalbach]

The topic of this thread is very disturbing to me. While I agree that hardcoding Alt.CC parameters would simplify things for those of us who are still struggling to understand exactly what the Alt.CC algo. does and how it should be used for "best" results, it also goes against everything that the free-source community stands for -- namely the ability to control every aspect of a program's execution.

For those of us who are not fluent in the language in which the codecs or applications calling the codecs are written, the "tweaking" ability has to be available through the user interface. (For instance, I'm great in FORTRAN, but not so good in C++. And no cracks about the FORTRAN please :D)

I fully realize that keeping Alt.CC parameters available to the user generates a lot of questions, discussions, confusion, and therefore work for those who wrote the algo. or for those who uderstand what the algo. does and how to use it. I very much appreciate those of you who are currently testing and trying to understand what effect, if any, the use of the Alt.CC has on encodes. I myself am still lurking and reading to try to understand what is being done by the Alt.CC code and how I should use it.

At some point, it is possible that the consensus of the XviD community will be that the Alt.CC algo. is not useful or that the "universal" parameters have been found and can be hard-coded. Until then, let's not throw the baby out with the bathwater. Please keep them open long enough for the community to get them fully researched and understood.

In the main Alt.CC thread, Defiler made the comment that a "White Paper" would be very helpful. He is so right. I am hoping to see something to that effect posted in the future once someone who has the time and understanding can get to it.

Thanks guys and keep up the good work! :)

[Kandor]

I havent done any xvid encodes yet, but I feel that I have to reply to the last message.

I think what Acaila ment was that we should hardcode some settings you can use instead of controlling everything your self for the users that dont want to or dont understand everything about it.
but there will allways be a userdefined mode so you can change everything your self and that makes the whole ide a very attractive ide for ppl that are new to this codec.
as me.

so I dont think you have to be negative about this.


thats it.


best regards

Kandor

[ckalbach]

Perhaps there was some confusion on my part regarding the meaning of the word "hardcode". I took it to mean that control of the various options would be closed to the user as opposed to "recommended values" or "default values" which would imply suitable values for most encodes while leaving the option available for advanced users to "tweak" for "best" results. :devil:

Thanks for the clarification.