How well does XviD handle solid coloured areas?

[JimmyBarnes]

Like walls and sky. With even DivX5.0.2, such areas usually show distinct blocks of colour instead of a seamless gradation.

Bear in mind that even some DVDs tend show slight "blockyness" in such areas.

Your thoughts please

thanx
JB

[Koepi]

Jimmy,

please stop doing this.

You're posting crap and I don't know why.
You should simply encode smoe movies with xvid and see for yourself - and not just ask really strange stuff. I'll strike you for rule number 4, netiquette.

Koepi

[soujir0u]

Hmm... I guess this problem is apparent with all lossy compression codecs. If you use a very high bitrate, and higher resolutions (512 horizontal and above) the effect will be a lot less noticeable. But like Koepi said, you should just encode and see for yourself.

[JimmyBarnes]

Originally posted by soujir0u
Hmm... I guess this problem is apparent with all lossy compression codecs. If you use a very high bitrate, and higher resolutions (512 horizontal and above) the effect will be a lot less noticeable.

The fact that it sometimes is seen even in the source DVD bears out your first comment.

I always use resolutions of 512(horiz) and above as this is the lowest I am prepared to go. I've done some 50 DivX3/5 rips. Following the 1st Pass % rule: the rip should be 60-75 % in size of the 1st Pass size (quant=2), I must say that even when I have used high bitrates (1st pass % around 75 %) I have not noticed that this has tended to decrease blockyness.

There are a few rips I have done where I would consider the problem to be almost non-existent - might have something to do with the quality of the source DVD. I remember trying to do a good rip of Jumanji, where the blockyness is really excessive and annoying. Eventually I found that much of the DVD had the problem.

Someone once mentioned that XviD did not have the solid colour blockyness problem. I didn't do anything about it then but after viewing my first XviD/OV/OGM movie (downloaded) a few days ago and seeing no evidence at all of the problem, I think I should have a serious look at XviD.

JB

[avih]

xvid's performance can be taken seriously for quite some time now ;)

[JimmyBarnes]

Originally posted by Koepi
Jimmy,
please stop doing this.
You're posting crap and I don't know why.

Koepi

It genuinely mystifies me what I have done wrong. I do not see why asking about the major problem which stops rips from being perfect should be labelled "crap".

http://forum.doom9.org/showthread.php?s=&threadid=25649 obviously has the same concerns.


You should simply encode smoe movies with xvid and see for yourself - and not just ask really strange stuff.

I will do that but as a complete newbie to XviD it would not surprise me if I don't get optimum results. If the codec is capable of overcoming this problem in experienced hands, then I believe it's reasonable to benefit from others' experience by asking this question.


I'll strike you for rule number 4, netiquette.
Koepi

You mean rule 2. With all respect, I feel your decision is unfair.

JB

[int 21h]

Walls and sky aren't really solid colored areas. Its the small variance and gradients that Video compression in general has problems with. Many times it doesn't devote enough bitrate to areas, and it smooths them out, or the areas appear to shimmer or jump, from what I can only assume is a poor motion estimation routine. (and/or bitrate allocation)

Anyways, there are several methods of 'cheating' to achieve better results and less fluctation with this type of material. The best is to filter spatial noise. Just removing some of the grain can help stablize those areas (be careful, the grain, a.k.a. sharpness, also provides 'detail').

In any event, its very very very subjective. You absolutely have to test this yourself because what may look good to me, may look like crap to you.

[JimmyBarnes]

Originally posted by int 21h
Walls and sky aren't really solid colored areas. Its the small variance and gradients that Video compression in general has problems with.

Agreed - you're spot on. "solid" colour is a poor term, but I think most know what it means if examples (like walls, sky) are given.

Many times it doesn't devote enough bitrate to areas, and it smooths them out, or the areas appear to shimmer or jump, from what I can only assume is a poor motion estimation routine. (and/or bitrate allocation)

Not sure about that. If one knew the cause, one could fix the prob.

Anyways, there are several methods of 'cheating' to achieve better results and less fluctation with this type of material. The best is to filter spatial noise. Just removing some of the grain can help stablize those areas (be careful, the grain, a.k.a. sharpness, also provides 'detail').

OK, I'll try that. First time I've ever seen it suggested.

In any event, its very very very subjective. You absolutely have to test this yourself because what may look good to me, may look like crap to you.

Saw my first XviD movie which I had downloaded a few days ago and it showed no evidence at all of the prob in those areas where DivX looks really YUK. I was amazed. I was hoping that XviD was markedly superior to DivX in this respect - maybe not, I guess I'll just have to try. But I value the experience of those who've been working with XviD for a long time - they could at least tell me if my quest is futile or not.

JB

[gldblade]

>If one knew the cause, one could fix the prob.

Easier said than done.

>from what I can only assume is a poor motion estimation routine

You make it seem like its totally bad! :) It's ok in most cases. It simply can't handle certain things. Such as gradients or fades. It's a limitation common in many motion estimation implementations (I think...)

>I was hoping that XviD was markedly superior to DivX in this respect - maybe not, I guess I'll just have to try.

It really depends on the movie. And how much time you're willing to spend tweaking the settings for better quality. Or how lucky you are in getting the best settings in the first place.

[Bulletproof]

I think the smoothness of gradients help when you have a DVD with a medium amount of mosquito noise and almost cleans it fully out without alot of consequence.

[-h]

Walls and sky aren't really solid colored areas. Its the small variance and gradients that Video compression in general has problems with. Many times it doesn't devote enough bitrate to areas, and it smooths them out, or the areas appear to shimmer or jump, from what I can only assume is a poor motion estimation routine. (and/or bitrate allocation)

It's because of the nature of the DCT and MPEG's quantization system - the resulting coefficients of a smooth gradient will have very small AC values, and altering them at all will change the gradient enough so that block boundaries become visible. There's not much that can be done about this, bar throwing large numbers of bits at such simple areas (to preserve the small coefficients) or designing some intelligent post-processing steps to disguise it afterwards.

Having noise dispersed throughout the area in questions should help alleviate the blocking problem, as greater AC values will be present, and over the course of a few frames the blocks should be pretty well disguised.

-h

[JimmyBarnes]

Originally posted by -h
[i]It's because of the nature of the DCT and MPEG's quantization system - the resulting coefficients of a smooth gradient will have very small AC values, and altering them at all will change the gradient enough so that block boundaries become visible.

That's what I was afraid of, that this is an inherent problem to
XviD and DivX5's compression schemes.

There's not much that can be done about this, bar throwing large numbers of bits at such simple areas (to preserve the small coefficients) or designing some intelligent post-processing steps to disguise it afterwards.

Targeting the (usually few) areas in question would be a problem I think. The codec automatically assigns more bits to fast motion scenes or can be configured to do so for low luminance scenes, but I can't see how it could automatically detect such "uniform" areas and act accordingly - am I correct?

Having noise dispersed throughout the area in questions should help alleviate the blocking problem, as greater AC values will be present, and over the course of a few frames the blocks should be pretty well disguised.

-h

What practical measures can I take?

thanx
JB

[tiki4]

If I am allowed I just like to share my experience:

I encoded the movie 'All The Pretty Horses' some time ago, there are some scenes with grey to brown stone walls in that movie.

I tried first DivX 5.0, then after a while DivX 5.01. Both performed poor in that scenes. DivX 5.02 performed better but is still way from perfect.
Surprisingly even without tweaking the codec too much XviD looked much better from the very beginning. That's just my experience with that problem and it was the main reason that stopped me from switching from DivX4 to DivX5.

The reason why I am still not using XviD on a general basis is that I like to wait till something like a beta release comes up. Also the implementatin of b-frames in DivX5 really helps with 1CD rips in my opinion. So I am eagerly waiting for the development to settle down a little bit and does no longer change its features on a daily basis (just like Vorbis did with its release candidates).

CU,

tiki4

[-h]

That's what I was afraid of, that this is an inherent problem to
XviD and DivX5's compression schemes.

Pretty much.

Targeting the (usually few) areas in question would be a problem I think. The codec automatically assigns more bits to fast motion scenes or can be configured to do so for low luminance scenes, but I can't see how it could automatically detect such "uniform" areas and act accordingly - am I correct?

The codec doesn't really know what it's assigning bits to. The nature of fast-motion scenes means both many bits used to store motion vectors, and many bits used to store the difference to the previous frame. Those put together mean big frames, and if a desired bit rate is set, such frames will be quantized heavily. Lumi masking is a bit different, where it actually bothers to look at the luminance values before deciding on a quantizer, but is still quite simplistic.

I suppose that gradients could be detected as well - the sum of the 1st 2 or 3 AC values, compared to the rest of the AC's summed, would get close (I guess? No idea actually). What to do afterwards might be tricky, as pathological blocking is often evident even at quant=2, though it is reduced a fair bit.

A nicer method might be something along the lines of TMPGEnc's "soften block noise" feature, though I still have no idea as to how it works.

What does this mean in terms of practical measures I can take?

Not much that I know of :)

I'd recommend using a deblocking post-processing filter that adds noise, it's been the most visually appealing combination in my experience.

-h

[JimmyBarnes]

Originally posted by -h
I suppose that gradients could be detected as well - the sum of the 1st 2 or 3 AC values, compared to the rest of the AC's summed, would get close (I guess? No idea actually). What to do afterwards might be tricky, as pathological blocking is often evident even at quant=2, though it is reduced a fair bit.

If you're correct, when a rip does appear almost free of blockyness, it probably has much to do with the exceptional quality of the source DVD.

A nicer method might be something along the lines of TMPGEnc's "soften block noise" feature, though I still have no idea as to how it works.

Sounds worth looking up.


What does this mean in terms of practical measures I can take?

Not much that I know of :)

I'd recommend using a deblocking post-processing filter that adds noise, it's been the most visually appealing combination in my experience.

-h

That sounds like DivX 5.0.2's Film Effect:
http://forum.doom9.org/showthread.php?s=&threadid=25649

thanx
JB

[int 21h]

Nando, and me, and Mythrndir (associate in crime that helped originally brainstorm for Nandub) sat around for many hours discussing this problem in general. Especially related to smoke or other ares of gradient color (i.e. Sandstorm in Stargate, Smoke in Gladiator, etc) We had some very very primitive algorithms in development to analyze the picture and throw more bits at it (basically that's what Mpeg-2 does), but we never really got very far or very involved with it.

IMO, the best way to deal with it, would be to somehow compensate for it (throw more bits at it!), and add it into the overall curve of the movie.

How this would be implemented isn't my department though ;)

Originally posted by -h
That's what I was afraid of, that this is an inherent problem to
XviD and DivX5's compression schemes.

Pretty much.

Targeting the (usually few) areas in question would be a problem I think. The codec automatically assigns more bits to fast motion scenes or can be configured to do so for low luminance scenes, but I can't see how it could automatically detect such "uniform" areas and act accordingly - am I correct?

The codec doesn't really know what it's assigning bits to. The nature of fast-motion scenes means both many bits used to store motion vectors, and many bits used to store the difference to the previous frame. Those put together mean big frames, and if a desired bit rate is set, such frames will be quantized heavily. Lumi masking is a bit different, where it actually bothers to look at the luminance values before deciding on a quantizer, but is still quite simplistic.

I suppose that gradients could be detected as well - the sum of the 1st 2 or 3 AC values, compared to the rest of the AC's summed, would get close (I guess? No idea actually). What to do afterwards might be tricky, as pathological blocking is often evident even at quant=2, though it is reduced a fair bit.

A nicer method might be something along the lines of TMPGEnc's "soften block noise" feature, though I still have no idea as to how it works.

What does this mean in terms of practical measures I can take?

Not much that I know of :)

I'd recommend using a deblocking post-processing filter that adds noise, it's been the most visually appealing combination in my experience.

-h

[-h]

Nando, and me, and Mythrndir (associate in crime that helped originally brainstorm for Nandub) sat around for many hours discussing this problem in general. Especially related to smoke or other ares of gradient color (i.e. Sandstorm in Stargate, Smoke in Gladiator, etc) We had some very very primitive algorithms in development to analyze the picture and throw more bits at it (basically that's what Mpeg-2 does), but we never really got very far or very involved with it.

Well there are a few ways, but the only way to replicate the gradient properly is to store the macroblock at quant=1, which isn't really a valid option.

In terms of XviD's current status, there are a couple things that could be done - remove the subtle thresholding that's performed (discards coefficients under a certain threshold) when such a block is detected (this would alleviate the problem at quant=2 somewhat), add random coefficient "noise" after quantization (to add an artificial film grain effect), or start playing with different MPEG quantization matrices.

In terms of actual detection, I can only guess that a given set of AC values (inside a DC range - we don't really care about blocks during extreme brightness) is fairly indicative of such a gradient, but we can only hope really. I'm not aware of any detection routines myself.

It's a rather large deficiency in current MPEG-4 encoders, it'd be nice to see some kind of reliable fix come about.

-h

[JimmyBarnes]

Originally posted by int 21h
Nando, and me, and Mythrndir (associate in crime that helped originally brainstorm for Nandub) sat around for many hours discussing this problem in general.

If nothing else, this makes me feel justified in raising this thread.

Especially related to smoke or other ares of gradient color (i.e. Sandstorm in Stargate, Smoke in Gladiator, etc)

Backlit areas of "gradient colour" (a much better term than "solid colour"), such as when one is looking through smoke etc, or underwater looking up, to a bright light source are the worse in my experience.

The XviD rip which so impressed me - "Legend" - had plenty of such scenes - wonder how they got no discernible blockyness? I viewed this under the same conditions that I view all my DivX rips.

It's a rather large deficiency in current MPEG-4 encoders, it'd be nice to see some kind of reliable fix come about.

As it's such an obvious deficiency, one can only presume it's a
difficult nut to crack :(

JB

[Uli]

Hello!

I think it's also depending on the source material. With a cleanly produced DVD, this gradient effects aren't as visible as on poorly mastered DVDs. One extreme example is "X-files the movie". Even on a very good DVD player you can see the very annoying blockiness in the scene in the beginning, where two men come out of a snow storm entering a cave or "Abyss" right in the beginning where the submarine comes out of the deep. If you encode these scene with XviD it looks even worse, because the encoder strengthens the effect.
BTW: Sunsets and sunrises are also well known scenes that produce this gradient effects.

Just my 2 cents...

greetz, Uli

[Bulletproof]

In my opinion you guys are nitpicking too much, XviD was never meant to be a lossless codec. There is no perfect codec, unless you're using a lossless one.

[int 21h]

Originally posted by Bulletproof
In my opinion you guys are nitpicking too much, XviD was never meant to be a lossless codec. There is no perfect codec, unless you're using a lossless one.

Yea, no sense in striving for highest quality or improved appearance. Why bother?

From now on, I am archiving movies ripped in HuffYUV

[soujir0u]

Originally posted by int 21h


Yea, no sense in striving for highest quality or improved appearance. Why bother?

From now on, I am archiving movies ripped in HuffYUV

Lol! The only time I use Huffyuv is when I rip PSX movies. Need a temporary file so that I can filter the crap out of them then encode to Xvid. But even at 320x224 res, Huffyuv takes about 100mb for a minute of video!

[Bulletproof]

Originally posted by int 21h


Yea, no sense in striving for highest quality or improved appearance. Why bother?

From now on, I am archiving movies ripped in HuffYUV

Yeah criticizing people who contribute really helps too. Instead of just whining like a ***** why don't you figure out how to add ***** gradients to your video yourself. And while your at it you can add support for progressive 2048x1024 video too. Your a ****ing nobody and just cause you have int 21h do you think your a higher authority than someone else? your nick means jack **** and your sarcasm is as ***** as you. Koepi was right when he said this thread is nothing but ********. BTW, you wanna archive movies with best quality then buy them you ****.

"Nando, and me, and Mythrndir (associate in crime that helped originally brainstorm for Nandub) sat around for many hours discussing this problem in general. "

Who the **** do you think you are? your a ****ing nobody, you try putting Nando's name next to yours and you think your a ****ing big shot.

"We had some very very primitive algorithms in development to analyze the picture and throw more bits at it (basically that's what Mpeg-2 does), but we never really got very far or very involved with it. "

Is that your solution to everything? throw more bits at it? yeah thats real smart. Your ****ing talking sarcasm about HuffyYUV and at the same time your taking compression techniques from MPEG-2 real smart.

(EDIT: Swearing edited by Nic)

[JimmyBarnes]

This might betray my complete ignorance of how MPEG compresion works, But...

Still images of the scenes in question (backlit smoke, sky etc) can be compressed substantially to JPG without blocky artifacts (at least for the sort of compression ratios typical for DVD->CD, say 4-8 or so), why (in concept) isn't it possible to devise a compression scheme which does the same for movies, which are after all a linked set of still images?

Just a thought
JB

Hmmm just had the thought: maybe the blockyness results from small differences between one image and the next, rather than being present in any one image

[JimmyBarnes]

Originally posted by Uli
Hello!
I think it's also depending on the source material. With a cleanly produced DVD, this gradient effects aren't as visible as on poorly mastered DVDs. One extreme example is "X-files the movie". Even on a very good DVD player you can see the very annoying blockiness in the scene in the beginning...

This is known as the GiGo principle: Garbage in, Garbage out

Jumanji where the VOBs/min were only 32 MB/min instead of the usual 40-60 MB/min for other DVDs, is another extreme example. Many DVDs show some trace of the blockyness problem, but to a much less extent than the subsequent DivX/XviD rip.

Conversely, some of the DivX trailers which are posted as shining examples of how well the codec works use sources with much higher resolution than the usual 720x576.

JB

[Bulletproof]

Jimmy, MJPEG codec exhibits the same macroblocks. The only way you can get an image to show fine details such as that that I have seen with JPG is to use Photoshop and practically keep quality to max which ends up making the file too big and at that point you would rather be better off using PNG compression. The consequence for having miniscule/extra fine detail in video is too large when those valuable bits could go to hard to compress frames. And I think even MPEG-2 offers higher colorspace also which helps to do such gradients.

[int 21h]

Originally posted by Bulletproof

Who the **** do you think you are? your a ****ing nobody, you try putting Nando's name next to yours and you think your a ****ing big shot.


You obviously have no idea who I am. So I'll forgive your lack of respect. You sure have alot of unresolved anger issues, take your profane comments away from here, and leave theorizing to someone who can express an opinion with out dropping the f word every 5 syllables.

In any event, I don't need to put my name next to nando's, he put my name next to it himself ;)

If you don't believe that any compression techniques from Mpeg-2 apply to its successor Mpeg-4, maybe you need to read up some.

Good day.

EDIT:swearing quoted from BulletProof's post edited by Nic

[kilg0r3]

hi there!

in the past few weaks, i have done a lot of test encodes, which had the consequence that i started seeing artifacts and ringing all over the place. to make a long stroy short i just couldn't contend myself with any of the encodes i had done. everything just looked terrible, and i asked myself if there head been any real progress regarding video compression in the past. :stupid:

This was the moment when i opened my cd drive and popped in a copy of galaxy quest which had been encoded using g'old divx 3.11 and flsk mpeg. --- i am a happy man again. the quality was so terrible motion scenes consisting only of blocks, which only were cleaned by the next keyframe (constant interval!! :eek: ). and still, despite the terrible quality, i enjoyed the movie very much. so when life gets all blocky get yourself a vanilla flask rip, watch it from beginning to end, and everything will be just fine.

ps: please respect the forum rules

[Teegedeck]

@Bulletproof: what did you think you were doing? not on this forum, my dear!
2 strikes for repeated violation of rule 4. And you force me to edit this thread, too, what I really hate to do - especially on this lame computer.

[int 21h]

I agree that we have come very far, and don't get me wrong, I admire the advances the XviD team has made, and I look forward to the future of XviD.

Originally posted by kilg0r3
hi there!

in the past few weaks, i have done a lot of test encodes, which had the consequence that i started seeing artifacts and ringing all over the place. to make a long stroy short i just couldn't contend myself with any of the encodes i had done. everything just looked terrible, and i asked myself if there head been any real progress regarding video compression in the past. :stupid:

This was the moment when i opened my cd drive and popped in a copy of galaxy quest which had been encoded using g'old divx 3.11 and flsk mpeg. --- i am a happy man again. the quality was so terrible motion scenes consisting only of blocks, which only were cleaned by the next keyframe (constant interval!! :eek: ). and still, despite the terrible quality, i enjoyed the movie very much. so when life gets all blocky get yourself a vanilla flask rip, watch it from beginning to end, and everything will be just fine.

ps: please respect the forum rules

[Nic]

Your not rubbing people up the wrong way again are you my fellow blackhat? :D

-Nic

ps
@teegedeck: well moderated...but when do we get to see pikachu as the animated gif? :)

[int 21h]

Originally posted by Nic
Your not rubbing people up the wrong way again are you my fellow blackhat? :D

-Nic

ps
@teegedeck: well moderated...but when do we get to see pikachu as the animated gif? :)

Wasn't trying to... ;)

[RadicalEd]

What the **** kind of aspect ratio is 2048x1024?
Sorry, couldn't resist :D

(EDIT swearing never actually present)

[kilg0r3]

@RadicalEd where to get a deseriousenizer? Could curretnly use one or two.

@ nic. don't you tell me that anigifs are allowed as avatars! :eek:

[soujir0u]

This thread kinda reminds me of Jerry Springer.

[FFiXx]

Hmm, interresting... Well I hope XviD will be better than divx3.11+nandub oneday, dunno when, dunno where. But I also really want to see how different it will be once B-frames r implented.

FFiXx over and out

[Aktan]

As far as I know, Xvid is betetr than 3.11+nandub right now :)

[Kyo]

@FFiXx
Don't get me wrong, you have tested XviD?

@Aktan
Sure, better and better...:D