Quants & matrices vs. blocks (3MB images)

[Richard Berg]

Rather than speculate on & on about the nature of high/low-bitrate matrices, etc. etc., let me just post some results from lots of constant-quant encodes of a 100 second (60fps) sample:

Huffyuv - 1.20GB
quant4-6of9 - 80.7MB (0.415 bits per pixel per frame)
quant4-6of9hvs - 78.0MB (0.402)
quant4-andreas78er - 71.0MB (0.365)
quant4-h263 - 57.4MB (0.295)
quant4-hvsbest - 56.4MB (0.290)
quant4-hvsgood - 51.9MB (0.267)
quant4-jawor1cd - 49.1MB (0.252)

http://richardberg.net/bin/temp/xvid-q4matrices1.png
http://richardberg.net/bin/temp/xvid-q4matrices2.png

Final stat for reference: quant4-h263 with all advanced features turned off (no qpel, gmc, trellis, vhq, or b-frames) is 88.9MB (0.458) and looks worst of all. I guess taking 4X as long to encode *is* actually worth it :)

----

My goal is to encode lots of this type of video to a reasonable bitrate. I love the look of 60fps sports, and we're a long way from doing MvBob in realtime, so "reasonable" to me means "very high" to most people.

However, unlike people encoding HDTV and such, I don't care about detail. Ok I care, but to my eye none of the encodes loses important detail. There's very little vertical sharpness in deinterlaced video anyway, so I'd love to save bitrate in this department to help make up for having 2x as many frames per second.

I do care about blocks. For whatever reason, these clips make Xvid generate lots of them. Some of those stills are quite ugly.

So, based on these data, I'm looking for a way to avoid blocks while keeping bitrate and detail reasonable. Let's assume the source Huff itself is already optimized for compressibility.* It appears I can choose a matrix like 6of9 and use high quants, or use H263 and hope for lower quants, or maybe another solution I haven't thought of. Comments welcome.

Newbie side questions:
- Are my bits/pixel/frame reasonable? What reduction should I expect the final 2-pass to be...50%? 90%?
- Why does H263 look better @ quant5 than quant4? (Look at the corruption in the upper-right of the picture). It doesn't on every frame, but this one isn't just a fluke.
- Why do lots of threads refer to Andreas and Jawor as low-bitrate? Unless I'm misunderstanding the concept, both are significantly higher bitrate than H263, and Andreas is actually higher than 6of9 (but with much worse blocks).

*I'm already satisfied with my filter choice: FFT3D + LimitedSharpen gives the nicest picture and the lowest bitrate of the options I tried, when holding all else equal. Cropping/Bordercontrol still need work but whatever changes I make there shouldn't affect my Xvid decisions.

(edit) Completely redid the comparison. It had major errors.

[Teegedeck]

Interesting post. :)

Some things that sprang to my mind:

Because there isn't much detail in the picture - and no noise either - you seem to be able to use SixOfNine with less bitrate penalty than in a run-off-the-mill encoding (compared to H.263) with subjectively better sharpness. So I'd say, stick with it that one. SixOfNine @ quant=4 certainly gives a solid, good compression/quality ratio.

But in the end it isn't about stills, and if you don't find a subjective difference between that one and Javor's matrix at a higher compression, why not opt for that one. High bitrate matrices are of course not meant for much higher quantizers.

It could be interesting to see some of those 'problem frames' - with shots from the lossless source, too.

Andreas' matrices were mid- to high-bitrate as far as I recall; Javor has made a low- and a high-bitrate matrix.

[yaz]

just some thougths popping in :

- if u don't mind fine texture but only blocking why don't u try some hvs tuned matrices ? say, hvs-good gives about the same compression as h263 but w/much pleasant look.

- 'high/low-bitrate-matrix' ... khmm ... one of the most messy term i've ever met on the forum. sure, the matrices u tried give (relatively) high bitrates at constant quant (compared to that ones which produce lower bitrates :) ) imho, a matrix itself tells (almost) nothing about the resulting bitrate as it depends heavily on the nature of the source.
andreas released 5-6 matrices, which one did u tested ? jawor has also some variants. his 2cd matrix is very close to the xvid (default) mpeg matrix, so u know, what u'd expect.
imho, the matrices u tested are not for that purpose u intend.

- i agree w/teegy. u aim typical action scenes (at least, i'd expect tons of actions when watching bb :) ), so stills wont tell too much. i would rather concentrate on action scenes.

- i would give more bits to such streams, but again, if u don't mind fine texture, it'd be ok.

the bests
y

[Teegedeck]

Originally posted by yaz
teegy :) ?
Certainly has a nice ring to it.

[*.mp4 guy]

From my experiance h.263 quantisization gives more blocks and less ringing then (almost) every cqm, or even the mpeg matrix. it also destroys gradients faster, so does ringing bother you more, or does blocking bother you more? on a side note matrices like 6of9 trade blocking and ringing both for texture degredation.

could I persuade you to try another matrix?

[edit] If you dont want to try another matrix stick with jawors...it looks less blocky to me then the h.263 frames and has a similar filesize.

[Sagittaire]

Huffyuv - 1478.1MB
quant2-H263 - 147.4MB
quant4-sixofnine - 99.4MB (0.442 bits per pixel per frame)
quant5-andreas - 89.3MB (0.397)
quant5-sixofnine - 81.6MB (0.363)
quant4-H263 - 71.2MB (0.317)
quant5-jawor - 66.7MB (0.297)
quant5-H263 - 57.3MB (0.255)


It's impossible to make comparison : if you want compare use same size for each sample ... for blind test, metric or other you must have the same size ...

- 'high/low-bitrate-matrix' for me high bitrate is matrix with better quant repartition for high quality encoding. For very high quality H263 could only use q2 or q1 (aarrrrrrggggghhhhhh ... !!!), high bitrate matrix like fox use q3, q4 or q5 for equivalent size. Visualy for high bitrate fox is not better than H263 but quant repartition (for I,P,B) could be very better for high bitrate matrix and quality visually more constant. IMO high bitrate is good for q2-q6 but useless with higest quant ...

[yaz]

Originally posted by Teegedeck
Originally posted by yaz
teegy
:) ?
Certainly has a nice ring to it. sorry for blasphemy. (no excuse, i know) i meant mr. (ms?) teegedeck w/my greatest and deepest respect i've ever been able to push out from myself. all in blue, of course :)

the bests
y

[Sharktooth]

The choice of a custom quantization matrix is made by taking 3 foundamental factors into consideration:
1 - Compressibility
2 - Final Size
3 - Desired Sharpness/Smoothness

There are also other factors (for cartoon/anime/CG etc.) but this is not the case.
So I cant understand this test. It's completely b0rked. There are no comparable results, no comparable quants (where's Jawor's @ quant2? and why there's no quant3?!?), no comparable filesizes...
Also b-frames are disabled. Did you know some matrices save bits on inter frame quantizations (so on b-frames too!!!)?
NO VHQ and no Trellis?!? OMG...

[Teegedeck]

Hm, I think you got one or two things wrong: Richard didn't intend this as a CQM comparison but asked for advice; and I believe that he probably has used Trellis etc. on his encodes, and the one without was meant as a low-quality anchor. Or that is how I understood it.

How about recommending one of your matrices for this case?

[niamh]

Because there isn't much detail in the picture - and no noise either - you seem to be able to use SixOfNine with less bitrate penalty than in a run-off-the-mill encoding (compared to H.263) with subjectively better sharpness.

I have a pile of Straight-Off-VHS-El-Cheapo-DVDs, and I can assure you that with prior adequate denoising, SixOfNine and hvs are highly suitable for the job. In fact, the results are simply amazing :) (And I'm talking one CD size encodes of 4:3 80's Z movies)

Are my bits/pixel/frame reasonable? What reduction should I expect the final 2-pass to be...50%? 90%?

Well that depends on the initial compressibility of the source, and as for the percentage expected, I find it also depends on the matrix (I haven't tested them all, by far). Personally, if I use H263, I find it needs 70-80% a least, while 6of9 would let you away with as low as 40-50% sometimes (that being with a comp test @ quant 3) << this is only my opinion.

Why does H263 look better @ quant5 than quant4?
I don't think it does at all :)

quant5-H263 with all advanced features turned off (no qpel, gmc, trellis, vhq, or b-frames)
I agree with Sharktooth there, and while you could argue that Qpel, gmc, and even B-frames are controversial, at the very least VHQ is very necessary.I wouldn't begin to consider it "advanced" and I wouldn't dream of using less than 4. I think you're depriving yourself of the biggest advantage of XviD :)

I would also say that I think the choice of matrix depends highly on the playback media, whether it be CRT, LCD or TV. I have a LCD pc screen, and all this influences my preferences (which used to be different when I had a CRT).

[Richard Berg]

It could be interesting to see some of those 'problem frames' - with shots from the lossless source, too.
I chose this frame because it has the most problems in it:
1) trail of white blocks behind player #4's head (upper-left of the frame)
2) the borders of the blue sideline get smeared (middle)
3) blocks in the dark areas of #30's jersey (middle)
4) stairsteps on #30's left arm (middle-bottom)
5) posterization & blocks around the black guy's head (upper-right)

None of these is a huge deal when viewed in motion (either on my workstation LCD or my living room CRT), but sometimes they catch your eye which is bad. Looking at #1 and #5 at 1/2 speed on my LCD is what made me say earlier that q4-h263 looked worse than q5-h263.

"storage\keep\bball-filtered-60fps.avi" is HuffYUV -- the source after AVS filtering but before Xvid encoding.

Richard didn't intend this as a CQM comparison but asked for advice; and I believe that he probably has used Trellis etc. on his encodes, and the one without was meant as a low-quality anchor.
Right. I'm looking for advice because this situation is different from most CQM threads: these videos have low detail but also very low compressibility.

Quick tests showed these vids would take up like 800GB at low quants, so I started by doing several tests @ quant 5. Quality wasn't very good, so I then did tried quant 4 with the CQMs that gave me the smallest file (H263) and best quality (6of9). At this point I had a lot of interesting data and little idea what to try next, so I posted a thread.

I used all advanced features (including lumi, forgot to mention that) on all 7 Xvid encodes shown above. The last test was just a sidenote to see how much extra quality & size improvement I was getting in return for 4X as much CPU time. The answer is "a ton" so I didn't post pics or pursue it further.

Andreas' matrices were mid- to high-bitrate as far as I recall; Javor has made a low- and a high-bitrate matrix.
In the big .zip I downloaded, the specific ones I used are called Andreas78er (lots of old threads talk about using it for low bitrate, if I understand correctly) and Jawor_1CD. BUT...I discovered that my filesizes were all wrong -- some of these encodes included PCM audio! I'll edit my first post, then the bitrates will make a lot more sense.


----


Personally, if I use H263, I find it needs 70-80% a least, while 6of9 would let you away with as low as 40-50% sometimes (that being with a comp test @ quant 3) << this is only my opinion.

So what you're saying is: even though 6of9 uses a lot more bitrate @ constant quant, I could probably make the final 2-pass encode the same size as I would with H263 but keeping the higher quality. ??

I will test that out shortly. I'm also testing some HVS matrices.

[Richard Berg]

Ok, I completely redid the comparison. Not only did I discover that some of my filesizes included PCM audio, but I found that doing crop/resize had a *huge* impact on compressibility. (I thought the fact that sports news bars are mostly static would make up for their sharp angles, but apparently not. Look at the bits/pixel/frame quoted by Sagittaire for comparison, though you have to subtract 19MB from andreas & jawor because of my audio screwup.) Thanks to this bitrate reduction, I used quant4 instead. I also added three HVS matrices.

The PNGs should be slightly easier for you to download now, too :) (664x400 vs. 640x480)

[Richard Berg]

Ok, time for more data :) Here are some two-pass encodes.

46MB (0.239 b/p/f) is the bitrate necessary to fit two whole games on a 4.7GB DVD. It represents about 50% reduction for q4-6of9hvs, 95% of q4-jawor.

60MB bitrate (0.312) would give 3.0GB files. 75% of q4-6of9hvs, >100% of q4-jawor.

I did the Jawor 1st pass @ quant3 using Teegedeck's zone trick; 6of9HVS 1st pass @ quant5.

http://richardberg.net/bin/temp/xvid-2pass-matrices.png

6of9 looks bad -- to me this is expected since it's having to use much higher quants; how are you supposed to get 50% reduction or more without losing quality?

Think it's worth trying 2-pass encodes with any of the "in-between" matrices?

[Ark]

Richard, do you know that quantizer scale using 6of9 is different than with other matrices, don't you?

For example:

Q1 H263, MPEG... = Q2 6of9
Q2 H263, MPEG... = Q3 6of9
Q3 H263, MPEG... = ~Q4 6of9

...and so on.

So at similar compressibility-ratio 6of9 will always have higher quantizers than other matrices, and in the first post 6of9 is always 1 "Q" foward the others (quality and size-wise), so it's not comparable with them.

[Sharktooth]

Originally posted by Teegedeck
Hm, I think you got one or two things wrong: Richard didn't intend this as a CQM comparison but asked for advice; and I believe that he probably has used Trellis etc. on his encodes, and the one without was meant as a low-quality anchor. Or that is how I understood it.

How about recommending one of your matrices for this case?
Well, since matrices are not comparable at the same fixed quant i dont think it's a good idea comparing results in that way.
As i said above there are some foundamental parameters you have to take into consideration when choosing a matrix for your encode.
There are also some conditions you may dictate (filesize, minimum acceptable quality etc.).
Now, once you choosen the desired filesize you can start trying different matrices and choose the one which looks better to your eyes starting from the matrices which meet the "foundamental parameters".
I cant recommend any of my matrices without knowing the compressibility, the resolution, the desired filesize and the duration of the clips.

[niamh]

6of9 looks bad

this is why we can't really help you; to me, SixOfNine looks marginally better than jawor's on these shots. I find less blocking,few more details preserved and the picture is crispier. To you, it looks bad. At the end of the day, you're the one who is going to watch them :)

So what you're saying is: even though 6of9 uses a lot more bitrate @ constant quant, I could probably make the final 2-pass encode the same size as I would with H263 but keeping the higher quality. ??
Yes this is my opinion indeed (anime excepted). You tested out that it isn't yours :)(though your second comparison does not involve H263)

[Didée]

Richard, I would not recommend 6of9 in this case, too. My blind guess from the very start was use Jawor's 1CD and let it go. (But then, perhaps, you would not have done all the testing) :)

Although 6of9 does not look bad to me on your last screenshots, I'm sure it has its issues when watching closely. Not so much on the side of detail, but I smell lots of blocks on rather 'flat' surfaces. Desired compression simply is too strong in this case.

The rule-of-thumb 50% compressability for 6of9 refers to a 1st-pass done @ quant 3. Referring to q4, 50% is too much.
If a 2nd-pass gets an average quantizer >5, for me that's high time to switch to another matrix, which usually is either Jawor's 1CD, or certain mods of 6of9.

[Richard Berg]

Richard, do you know that quantizer scale using 6of9 is different than with other matrices, don't you?
Yup. That's why I did its first pass @ q5 and the others @ q3.

If a 2nd-pass gets an average quantizer >5, for me that's high time to switch to another matrix, which usually is either Jawor's 1CD, or certain mods of 6of9.
In the 46MB test, it's actually quants 7-10! Surprising it doesn't look worse than it does.

I cant recommend any of my matrices without knowing the compressibility, the resolution, the desired filesize and the duration of the clips.
All of that info is in the thread, but I'll repeat it here just to get it in one place.
Compressibility: I'm not aware of a standard measurement, but there are several constant-quant bitrates in the 1st post that should give you the right idea. Consider it a worst-case; I'm still messing with stronger AVS filtering via DePan.
Resolution: 664x400 @ 60fps. (I'm considering dropping it to 640x384 or 600x360).
Filesize: Don't care too much since I use large RAID, so long as I'm not outright wasting space. 1/2 DVD would be convenient for mailing full-quality copies to friends, though I might have to recompress to SVCD for compatibility with non-geeks anyway. The most recent tests were for 2.3GB and 3.0GB (figuring 50MB of audio @ ~80kbps).
Duration: most are 80-85 minutes

I find less blocking,few more details preserved and the picture is crispier.
We agree that it's subjective. FWIW, here are the differences I see:
Stills - detail is about the same; about the same number of blocks, but they are more defined (harsher gradients) in 6of9hvs; slightly less noise in jawor
Motion - imperfections in the floor texture jump out at me less often when looking at jawor; same total amount of aliasing on the sidelines when camera moves, but 6of9 aliasing looks "jerky" while jawor looks "wobbly"