Are you experiencing ugly macroblocks and scretches in dark areas?

[moshem]

Hi,

just wanted to check if I am the only one who has this problem.

I encode high quality sources using Divx 4.12 and get exceptional quality in all parts of the video except on black/dark areas. This seem to be some sort of a bug or missing feature from the codec and it looks so bad.

This is specially noticable in movies like Gladiator and Lord of the rings where darkness rules over large part of the movie.

If I watch the video on my monitor - I can hardly notice it but it's there (the contrast is too low) but when using my TV Out (high contrast) it is very noticable and very disturbing.

I tried it all , higher bitrate , Temporal smoother, Smart smoother - nothing really helps removing these.

Is it just my computer doing that ? or do you also experience this problem ?

I wish DIVX guys would do something in the codec in order to solve this issue even if it means increasing file size a little.

This problem takes a great encode almost comparable to DVD quality and throws it back to the age of streaming video on 56K , just looks awefull.


please let me know what you think.


thanks

[obelix]

Had this problem yesterday on a notebook. Changed the display settings to 32 bit and the blocks were gone.

[rui]

Originally posted by moshem


This is specially noticable in movies like Gladiator and Lord of the rings where darkness rules over large part of the movie.

thanks

how did you get Lord of The Rings movie???
I would like to buy vcd or dvd.

[Kandor]

my guess is that it is the svcd that he is reencoding to divx

[UHT]

lord of the rings isnt out yet....officialy, most svcds of it knocking around on the net are from cams or screeners

[pia2001slo]

You're wrong. I have perfect DVD rip (Divx) on 2CD's with
subtitles.

Jolly

[rui]

Originally posted by pia2001slo
You're wrong. I have perfect DVD rip (Divx) on 2CD's with
subtitles.

Jolly

Did you make it yourself, or did you get it in some site??
Because i just went to amazon.com and the only LOTR dvd they have listed was a movie made in 1978. :confused:

Sorry for not sticking with the original thread subject, but i just loved LOTR, and would like to buy it.

[moshem]

AS for the blocks solution by moving to 32 bit , I did it myself and it did not help.

What kind of a notebook are you using (manufactor and model)? What operating system ? resolution ?


thanks

[obelix]

Toshiba Tecra 8200 Win2k 1024x768 32 bit

[moshem]

there ? graphic card memory ?

are you running XP ?


I am using a IBM ThinkPad T21

[obelix]

Trident CyberBlade XP with 16 MB. Running on Win2000 as stated above.

[moshem]

I just tried 640x480x32 and the blocks are still there!

You are saying that you had big blinking/non-blinking blocks over dark areas and once you switched to 32-bit they are completly gone ?
you are not experincing artifacts at all in dark scenses ?

BTW,

Are you watching your movies on the TV or on the monitor , TV will show the much more of the artifacts

[obelix]

The blocks were not blinking, but the were off colour. It was an almost black cloak in a cartoon. Brighter blocks moved over the cloak. These are completely gone. I did not try on TV. The Notebook has a TV-Out but it't not mine and we dont't have TVs in the office.

[moshem]

these are exactly the blocks I'm talking about, my guess is the if you use your TV-Out you will find out that many of these blocks are back! (the tv has a much higher contrast which emphasis these blocks that are impossible to see on the darker monitor)

[Doom9]

@moshem: don't worry.. every codec has these artifacts (have you read my codec comparison?).. it's one of the issues that I constantly mention to the developers and ask them to fix it.. dark parts just look bad and there's nothing you can do (maybe filter the hell out of it but for me any filter but the resize filter is a big nono)

[moshem]

how are you ?

If you recall , months and months ago when DIVX4 was opendivx and the DARC guys were working on the beta versions , you and I and many others kept feeding them with problems to be solved.

I for one, constantly reported problems with Matrox G400 , IBM thinkpads and huge ugle macroblocks in dark parts. although some of issues were addressed , some were just ignored.

as far as I know , DIVX4 people know about the dark macroblocks for more than 6 months now but they never addressed this crusial issue. I can't really understand why?! in my view , the top-two features that prevents DIVX4 (MPEG4) from totaly making MPEG2 obselete are :

1. Take care of the damn dark macroblocks
2. Optimize the decoder to make use of Pentium III SSE (there quite a few millions of this still in service) , it is not possilbe to play 640x480 clip on an average graphics card & P III 800 without dropping post-proccesing.


You can create wonderfull looking movies using DIVX4 - in most scense almost as good looking as the original DVD - until you get into a dark scene, this is when DIVX4 looks like those 56kbps streaming solution common 5 years ago. In my mind a fix for this problem should be top priority as every day goes by more and more awefull quality rips are being made and distributed.

As I mentioned before I recently got "Lord of the rings" rip and half of this movie looks so good and so sharp & colourfull - the other part in the swamps and caves and nights scense look so bad - you just can't believe the same codec was used for both parts. Now I have to decide if I want to keep this rip or not , tough choice!

Doom9, if you asked the development teams to do something about it , maybe you got the inside track , are they working on it ? is there a chance to see a solution for this soon ?


thanks

[UHT]

there is NO lord of the rings dvd rip, the dvd isnt even finished yet

[rui]

Thanks for clearing this UHT

[moshem]

There is a 2CD DVD Screener rip of LOTR out there - it is very new and was distributed on a limited basis - I got mine from a friend, the quality of non-dark scense is excellent and no doubt were made of a DVD and not an SVCD reencode (the resolution is also bigger then the SVCD version).

now that this is clear , please stick to the topic of this post, I think this issue is very serious.

thanks

[obelix]

Did some experiments this evening. No TV, the notebook can't be taken home to test. I increased contrast on my 19" to do some encodes. Seems to be a quantisizer issue. Please try a 1pass encode with minquant=maxquant=2 (or 1pass VBR with quality 100%) and tell me if the blocks are still there.
The observation is that at large black or dark areas the codec thinks it can compress very well which gives you too high quants. If the blocks are gone with the above settings I can offer a solution for your prob.

[moshem]

and that is how I got to the conclusion that is is a codec BUG or missing feature.

I did :

A. Encode 1 minute of dark scene , 1 pass , slowest , bicubic resize , bitrate 2500 , min quant 2, max quant 2.

Result : excelent picture quality, ugly big macroblocks and scretches in dark areas.

B. Encode 1 minute of dark scene , 1 pass quality based , 100% , bicubic resize

Result : excelent picture quality, ugly big macroblocks and scretches in dark areas.


There is no way around this! My theory is that the codec looks at a relativly big area that is black or semi black and "says" : "good, this is an opportunity to reduce file size" and it then raise the quantizer to very high value and compress , now while encoding very minor changes occur inside that black area and becuase of the high compression ratio , every change is reflect in a big block which is not of a pure-black colour. just my theory..

The problem is that even if specificly told to , the codec ignores the 2-2 quantizer and raise it - no matter what.


BTW, if you can't watch it on tv , increase brightness and also play with the contrast. It is also to see those if you view your video full screen rather than in a window.


let me know what you think

[obelix]

OK, you are right. When I increase brightness AND contrast to 100% I find those blocks. They did never bother me at normal settings. So those blocks on the notebook seem to be another issue because I got rid of them. They were much larger, too.

[moshem]

I too , can't really notice them on my monitor at normal monitor brightness & contrast and it does not bother me at all.

Imagine my suprise when I first watched my newly encoded movie on my TV , my previous "perfect" rip looked so bad I couldn't believe it.

If you think the macroblocks you saw on your monitor when you increasted brightness are bad , wait for the TV , it is much worse than a brightened monitor and this is why I am so worried.

Imagine 1-2 years into the future when everybody will have a video card that has TV-out or better yet maybe there will be dedicated players. You will want to view your precious movie collection on your TV and suprise suprise all of your collection happens to suck! now imagine how much work and time you spent on making these rips is worthless - you thought you have semi-DVD backup of your beloved movies but with this kind of quality - who wants to view them at all! now we are back to rent the DVDs!!

get it ? this is why I think it should be resolved ASAP.


let me know what you think

[kvs]

This may be a stupid idea, but what about using tv scale instead of pc scale in dvd2avi? Just a thought.

And is it just as bad with xvid.. ?

And while I'm at it, does anyone have an idea on when
the next version of divx will be out?

Also, the thoughts on the collection sucking in the future
when viewed on TV because of the macroblocks: won't perhaps
our TV's use a technology more similar to our PC-monitor in
the future, and thereby reducing this problem?

Don't know, just a newbie =)

[moshem]

** This may be a stupid idea, but what about using tv scale instead of pc scale in dvd2avi? Just a thought.

== Does not matter

** And is it just as bad with xvid.. ?

== I did try it tonight , and sadly it has more or less the same problem which is no suprise as it is based on the same code as DIVX4


** Also, the thoughts on the collection sucking in the future
when viewed on TV because of the macroblocks: won't perhaps
our TV's use a technology more similar to our PC-monitor in
the future, and thereby reducing this problem?

== No , televisions will always use different colour tempture and higher contrast & brightnes values , even if the future TV will be digital (which will probably not going to replace analog TV in 1-2 years) . So yes , we will be stuck with a collection that is only viewable on our computer, on the TV it will look great until a dark scene comes along - then it will suck!

[kvs]

So perhaps the only solution is that they DO fix it,
and fix it soon! So anyone know anything about the
next release? Like when's it coming, and if it will
improve on these problems?

And since that new tv-out card with divx-
decoding is coming soon, shouldn't they make an effort to release
a new version by then that actually fixes the problem,
before everyone _sees_ the problems? :sly:
They're like full of hype, and it would do them no good
if this problem is not fixed considering they're making
an effort to make more people watch divx on their television
(or so it seems.)

Plus, any email we can mail to and ask for a fix? Like
a wishlist or something... perhaps do a petition...
and someone HAS told them about the problems, right... ?

Who knows if the divx4-developers even _own_ TV's! :D

[kvs]

Oh yeah, and has someone discussed this with the
guys at xdiv? Maybe they're more willing...

[Foo]

Originally posted by UHT
there is NO lord of the rings dvd rip, the dvd isnt even finished yet

Actually there are a couple of them. It's a screener DVD, so it has an occasional "Property of New Line Cinema" message, but it *IS* from DVD, 5.1 DD sound and all. Screener DVDs are quite common nowadays.

First rip was by TCF, a four disc SVCD set. Some people made two disc DivX conversions of this.

TCF later gave their copy to DVL, who made (and released) a two disc SBC DivX conversion. The also made a three disc SBC with the DD 5.1 track which wasn't released publicly.

Now back on topic. This is common among all mpeg4 codecs. DivX4 tends to do a better job in dark scenes than the old codec IMO (and to a lesser extent so does Xvid). You shouldn't really be amazed at the output differences on a TV compared to a monitor when you think about how different the displays are. It still looks excellent considering how much compression is taking place. I don't think the developers of any of these codecs are "ignoring" complaints about this. There is only so much they can do. If you really are that upset, grab the Xvid code and come up with a better solution :D

[moshem]

I'm not upset , I just want to help & improve DIVX for all of us.

the problem is common to all MPEG formats and not just MPEG4 , it is not an issue of good compression since even faced with the highest bitrate(6000) and lowest quantizer values (2-2) - DIVX4 still produce these blocks - unlike MPEG2 which upon given the freedom it needs (and it needs a lot!!!) the problem will go away.

I am no expert programmer in the video compression area, so I can't really do it myself , I do what I can by helping debugging and improving the codec.

If it is a limitation that cannot be overcomed , I wish they say so so we will know that this is a technological limitation. Personally, I doubt , if they can do complex scense soooooo good , they should not have any problems with simple area like shades of black.

[kvs]

I'm not upset personally, I'm a student who don't even own
a TV... just have a 21" monitor I got from an office, and well..
it's big enough for me at the moment. :D

I might get upset when I see my divx files on
television one day. Don't really know what to expect.
You all make it sound like it's hell on earth or something,
that it looks like a 56k stream... it can't be THAT bad, can
it? I mean, I watch a lot of old VHS damnit!!! Can it be
worse than that? (I hope not!:devil:)

If these issues simply can't be fixed with current technology
like divx... what if someone DOES create widescreen television
that behave exactly like PC monitors.. ? :cool: Problem
fixed.. sort of. Of course, I can't imagine that happening any
day soon, that our PC's will be attached to 32" widescreen
monitors, but.. oh well.

Another thought I had... does (s)vcd suffer from this problem too?

[kvs]

"I'm not upset" :D :D :D
LOL! That was funny!

"I wish they say so so we will know that this is a technological limitation"

I actually wrote info@divx.com something along what you said there... that if it IS a technical limitation, at least let us know damnit...

of course, I don't know WHO's gonna read that mail, if anyone at all...:rolleyes:

[moshem]

"the 56k streaming" saying was just meant to make a point. It's bad but not that bad.

what happens that in dark areas insted of getting fine details you will get large blocks in shades of black that sometimes blink and sometimes appear as "holes" in an darker backgroud ,this can really let you down and make watching a movie in virtually DVD quality - problematic, it's like a wake up call saying "hey, it's not DVD , don't forget".

[moshem]

I just want DIVX to be better and make those disappear!

BTW,

I just noticed that there are others who reported this problem :

http://forum.doom9.org/showthread.php?s=&threadid=16195

[MaTTeR]

After using the various builds of XviD for about 2 weeks now I've found that a 2ps without Lumi-Masking reduces the number of blocks in dark areas. Maybe it's just the different movies I've encoded but it certainly looks better than the blocks I'm getting in Div4. Just my 2 cents:)

[kvs]

Sigh... I would've liked to do some comparisons between
xvid and divx4 with "dark city", but that's not going to
do anyone good since I don't have tv-out access.. argh.

And saw the thread you created at divx.com too just a
minute ago, interesting as well. Too bad noone from the
divx team is supplying us with some information..

Maybe it will be easier to get some answers from the guys
at xvid.org? Surely some of you must have asked the xvid.org
developer whether it's possible to improve on this issue
or not? and perhaps even got a reply... ?

[moshem]

just now but XVID isn't ready for trusting it with my rips , it is too young & buggy... but if they can fix it...

[Doom9]

a reason why I added the chapel scene of SPR to my codec comparison were exactly these blocks.. they're there with every codec I tested and every codec I ever tested for that matter.. so I guess each codec maker things as long as the competition doesn't do anything against it we won't have to invest into R&D either.

can't tell if DXn or the XviD team will ever solve this one.. you certainly can't expect anything from DivX3.. that is a dead end and I wouldn't be too surprised of this year means the end of it.. plus.. the "official" scene approved DVDRip (Screener rip to be precise) is of course done in DivX3, not DivX4.

[moshem]

but I have made by share of rips and DIVX4 suffers from the same problem.

Doom9 , as I remember, you have good relations with the DIVX development team, can't you urge them to fix this irritating problem?

BTW,

MS is going to demostrate Corona ,the next version of windows media next week, so I hear , and they took one great challenge on themselvs , they are going to use Corona in movie theater to show a movie in High Definition!! format as decoded by media player and the new video codec and a 5.1 audio codec.

I wonder if they solved the ugly macroblocks in dark areas problem in their codec before taking on this huge challenge?! my guess is that people in the theatre will not be very happy with ugly macroblocks when the movie gets dark , not when every macroblock is 1 feet in size ;-)

[kvs]

Mmm yes the next windows media encoder... wonder if it will
be the bomb or not. Sounds rather good, but of course, everything
microsoft says usually sounds "rather good". :D

and what if they HAVE fixed this blocking-issue... mmm..
argh I hate the present... good things always seem to
come in the future... uhm.. yes. :)

[Foo]

Originally posted by kvs
Sigh... I would've liked to do some comparisons between
xvid and divx4 with "dark city", but that's not going to
do anyone good since I don't have tv-out access.. argh.


Actually this is a bad example. DC compresss extremely well, and the artifacting isn't bad at all. The part that looks the worst is probably while the camera descends through the fog in the very beginning, but that is to be expected. A 640x272 one disc with r3mix'd audio looks very good via Tv-out.

Pitch Black is a pretty good test. The combination of darkness and odd lighting (from flares, torches, etc.) shows the effect pretty well. Looks great on a monitor, a bit blocky on a bigscreen.

Any movie with shots of empty space (as in outer) shows it very well too. The extemely dark black just doesn't look great on a TV. Something like 2001 would probably be a good example.

Some light smoothing with SmartSmoother/TempSmoother/etc helps out a bit without degrading crispness too much.

foo

[gldblade]

>MS is going to demostrate Corona ,the next version of windows media next week, so I hear , and they took one great challenge on themselvs , they are going to use Corona in movie theater to show a movie in High Definition!! format as decoded by media player and the new video codec and a 5.1 audio codec.

>I wonder if they solved the ugly macroblocks in dark areas problem in their codec before taking on this huge challenge?! my guess is that people in the theatre will not be very happy with ugly macroblocks when the movie gets dark , not when every macroblock is 1 feet in size ;-)

If it's anything like their previous codecs, they'll just blur away some of the details in an attempt to get rid of macroblocks and hope everyone won't notice...

Of course, that would be kinda difficult on a big screen...

[buba king]

Originally posted by UHT
there is NO lord of the rings dvd rip, the dvd isnt even finished yet

I'm sorry but you are totaly wrong! There is a SVCD DVD Screener rip that has near DVD Quality! Its a 4 cd rip so i guess its kinda hard for n00bies to get since they have to rely on Kazaa and HTTP Warez. There is also a re-encode of this rip, DivX, that has terrible quality IMHO. I have The SVCD myself and i'm inpressed with the quality... But 4 cds is overkill :-).

www.vcdquality.com has caps from the Rip. (http://www.vcdquality.com/index.php?page=image&spath=1&id=695)

Please Note:
I will ignore any requests :P

[rui]

Originally posted by moshem
There is a 2CD DVD Screener rip of LOTR out there - it is very new and was distributed on a limited basis - I got mine from a friend, the quality of non-dark scense is excellent and no doubt were made of a DVD and not an SVCD reencode (the resolution is also bigger then the SVCD version).

now that this is clear , please stick to the topic of this post, I think this issue is very serious.

thanks

You are right.
I want to apologise myself, like a said above, for not sticking to the subject thread.

[-h]

Just a few points:

You can't optimise an MPEG4 decoder with SSE instructions, because there simply aren't any SSE instructions that are useful for the data being processed. Sure, you could try rewriting the codec to work with floating point numbers, which are all SSE can process, but then it'd run at about a quarter of the speed.

Also, I haven't experienced this black block issue but it seems as though it's to do with either:

- The nature of the DCT transform itself with dark input (unlikely, given MPEG2's quality)
- The YUV colorspace being used lacks the full 0-255 value range, thus decimating the extreme dark-end of the range (perhaps more likely, though the YUV space being used in XviD is 0-235 I believe, thus preserving the darkness). It might be interesting to see what a full 0-255 colorspace implementation does to this issue. If this is the case, it could possibly be alleviated at playback.

I'm honestly just guessing, and won't be able to experiment until the weekend at any rate.

-h

[ulfschack]

LOTR discussion apart ...

I've had these "macroblocks" appear everytime i've tried porting a film from a computer to a TV. I've tried on three different graphics cards and while the severeness of the blocks varies they still remain. Now, not a single post (albeit based on rapid browsing) has suggested anything in the direction of it being a tv-out problem inherent in the g-cards.

Well, it is!

Here's a simple test that should eliminate any doubt.

Take a pariticulary dark and ugly portion of the divx. Code it to svcd (mpeg2 480x480 or 480x576 depending on region). Burn it to cd with Nero (no need to use more advanced authoring to prove this point). Pop it into a stand-alone DVD player and watch.

The quality SHOULD by all means be degraded since we've now compressed the video twice, but you see a dvd player is an mpeg-tv-expert having no trouble converting from rgb/YuV to the frequenzy domain as do most (all) tv-out cards. What I'm saying is that now you might see REAL macroblocks, but the big ones in the dark areas (lets call them Luma blocks) are basically gone. QED.

It doesn't take long, so just try it out and write back

cheers

PS can I have a copy of those LOTR cd:s :)

[moshem]

I tried it on different machines with different g-card and different TV-out :

1. IBM Thinkpad T21 w/S3 Savage IX TV Out (one of the best quality TV-out out there)

2. Matrox G400 TV-Out

3. Sigma Designs H+ via DivxPlus (excellent quailty TV-Out)


all exhibit the same problem. So I dont think it's a specific card driver/hardware problem.

Reencoding it to SVCD will just blur everything since you took if from high bitrate (DVD)to low one (DIVX) and from the low to an high again (SVCD). There will be a major detail & sharpness lose, so I would not be suprised if some of the blocks will be blured a way as well. Also the colour conversion might be also responsible for reducing the artifacts as the quality degrades.

But, it might not be the encoder at all , as we can't take a part the encoder & decoder. The problem might come up in the decoder , it could be that the encoder does a good job , and the decoder is screwing it up - if that is the case , this is excellent since once fixed, our movie collection will be worth a lot , but if it is an encoder bug , then....

[ulfschack]

This is exactly what I'm saying. It's the tv-out cards. You say they're good without even trying to make an experiment. How can I win :)

You go on assuming that coding to svcd will blur the picture and the alledged macro blocks. And I counter with: 1) Notice how these Luma blocks are bigger than ordinary macro blocks (definitly NOT 8x8) and sharp as razors, so they'll appear nice and big no matter how hard you compress. 2) Go to svcd from the original DVD allready. Geez ... just try man.

cheers

[moshem]

I am saying these cards are of excellent quality becuase of their performence regardles to this issue.

I did not try to covert an DIVX to SVCD , but I did see the original MPEG2 in mediaplayer on the same TVout and it was without flow, I also made a lower bitrate version of MPEG2 from the source , it exhibit blur but *NO* macroblocks , so if we eliminate the TV-Out we are left with the encoder or decoder.

[Doom9]

the DXn people have asked me before what sources they should test new builds on.. so I gave them a bunch of titles and scenes, including such dark scenes, underwater scenes and other problematic stuff.. rest assured they are aware of the problems but nobody can tell at this moment if it's going to be fixed in a future release. And it definitely has nothing to do with TV out because I've had these ugly blocks since I started ripping and I never watched on anything but my pc monitor.

as for XviD.. I'm sure some of the developers have seen my article and are aware of the problems too, I even submitted a sample clip from the chapel scene to them.

[moshem]

can you refer them to this thread , so they can see how disturbing it is for lots of people. maybe that will make them take it more seriously ?