Why is luma masking recommended?

[Franko30]

Hi,

while reading Doom9's guides and reading through almost all XVID related threads I noticed that everybody is happy with the "use luma masking for 2nd pass" recommendation, but nobody ever asked

a) what it does
b) why it seems to be necessary.

While playing around a little bit with XVID encoding I didn't notice any visible difference between luma masking on or off...

Could anybody please be so kind to explain this to me (and all the others who don't dare to ask in case they might sound stupid)?

Thanks

Frank

[Teegedeck]

...and hello to you.

Lumi masking is a first 'psychological' innovation in XviD; it is supposed make use of the fact that the human eye tends to notice encoding errors less if they happen in very dark or very bright parts of the picture. XviD is - in contrast to DivX - capable of using different quantizers for each macroblock. Lumi-masking compresses very dark or bright areas stronger than medium ones. So it will use less bits on some frames in the second pass than in the first pass. The saved bits are of course spent again and that way we gain a bit of quality in the medium-brightness part of the picture. It's a trade-off, but one without grave consequences.

BTW: You're using a Celeron 1300? How many fps do you get? (Just being curious.)

[Franko30]

Around 10 fps (XVID) using Virtual Dub Fast recompress (together with the *.avs files created by GKnot) from 704*576 to 576*432 using normal bicubic resize & Smart DeInterlace (Interlaced Source). XVID search precision 5 and H.263 quantization.

On my other encoding machine (Athlon Thunderbird 1400, DDR RAM) I get around 14 fps (XVID) using the same encoding setup.

I use the Celeron 1300 with Topgrade's Slot1 VRM Adapter in exchange for my old 933 MHz CPU as I wanted a little more speed but didn't want to buy a new MB and new RAM and a new processor plus reinstalling the whole system; maybe later this year...

But considering the speed differences to the Athlon, I'd say it's not worth it upgrading the way I did with the Celeron system. Hey, but MHz counts... somehow.


Back to the topic:

If some bits are stolen from bright AND dark parts, doesn't this lead to making the "Macroblocks in dark areas" issue even worse? I read something like that (dark areas getting better without luma masking) in

http://forum.doom9.org/showthread.php?s=&postid=76959#post76959

posted by Peters.


Frank

[-h]

It will accentuate the dark block problem, but until the actual cause is found, I wouldn't worry too much about it.

The solution will probably just be adding random noise to extreme dark areas in the post-processing.

-h

[kastro68]

Don't use luma masking ever!

I have turned it on for the second pass and it is not worth the degradation in picture quality.

Apart from the irritating pink shadows...it makes the video just too blocky, especially with writing on a dark background.

[Franko30]

It's always nice when somebody confirms what my own eyes saw on the monitor as well as on the TV:

From Saturday to Sunday I encoded 2 Voyager episodes and one Farscape episode all without luma masking and it really looks better (especially in dark parts) than with luma masking turned on for the 2nd pass.

Just as "Peter", whom I cited above, found out earlier...

And as koepi's binaries, that I tried earlier on Saturday night, turned out to deliver really blocky results,:confused: I'll just stick with nic's February 2nd binaries.

Frank

Edit: Now, as the blockyness issue seems to be solved in almost all binaries, it's "free choice" again. :D