XviD with BLACK Solid Background looks Terrible !! [Archive]

bobby8798

9th February 2003, 04:19

Tried encoding 1-pass XviD with colorful moving objects in the foreground against a PITCH BLACK Background

used 3000 kbits (increasing bit rate has zero impact)
changing i-frame quantizer has zero impact (even tried 1-2)
using latest koepi unstable build XviD-02022003-1
tried h263 and mpeg
1 pass CBR, or 100% quality, or quantizer=3
lumi-masking OFF; on is just as bad
CBR smoother from 0-100, no impact

compared this to DivX3, and DivX is far superior than XviD !!!

the XviD encode has a lot of noise in the background which looks really horrible. looks like color from the foreground is leaking everywhere. Extremely blocky color leakage !!!
The divx encode looks very clean. It has only about 10% of the XviD noise.

If the background is lighter gray, the situation REVERSES and the XviD is far superior than the Divx. It seems like XviD has a very serious limitation against SOLID DARK colors ???

anyone know how to fix this?

9th February 2003, 05:38

Use a filter to adjust the levels so that you have more black. Does 2-pass help?

Zarxrax

9th February 2003, 07:41

Maybe you could post a clip to show what you are talking about?
I don't think xvid has any trouble with black, as I just created a test video myself to see if I could cause some trouble in the codec (pure black background with colored text zooming all over the place) and for all quantizers below 8, it looked very nice.
Probably you just need to filter your source some, maybe adjust the levels like Y7 said. Spatial and temporal cleaners might help as well.

Teegedeck

9th February 2003, 12:20

Yup; post a screenshot, please.

Koepi

9th February 2003, 13:45

Or even better: search the avisynth forum for "luma offset".

Use marc's mpeg2dec3.dll and add this to your avisynth scripts:

LumaFilter()
UnFilter(-5,-5) # - is smoothing, + is sharpening

Regards
Koepi

Sigmatador

9th February 2003, 13:49

maybe you should use the refdivx's lumimasking code (koepi's build ^^)

great for dark scene ^^

edit: oops just seen: " just as bad" ---> :confused:

iago

9th February 2003, 14:39

Listen to Koepi. That issue was discussed extensively in the past and a working solution was finally proposed/achieved in the "Lumi Masking" thread in this very XviD forum.

regards,
iago

Use LumaFilter(-2) / LumaFilter(-3) / LumaFilter(-3,0.8) etc. depending on the source, together with UnFilter.

Lothar

9th February 2003, 16:27

Thanks for the info, it's the same problem I posted about some time ago ;)

Now I'm gonna test this solution, thanks again :)

bobby8798

9th February 2003, 20:30

LumaFilter cleans the noisy blocks !!!

It tends to brush out and use one standard color to remove the blocks and noise.

LumaFilter works nicely to remove the noise in a dark background. I will try a setting of (-1) instead of the default of (-2) because I don't want some of the fine details to have its color transformed to black. LumaFilter also tends to add more of those tiny white DOTs.

I didn't try the UnFilter, because it was not part of the avisynth v.250 package.

Thanks to everyone for being so helpful

angelyote

9th February 2003, 23:24

Here's the thread everyone's referencing.
It's well worth reading and is a who's who of doom9 personalities. :)

educate me (http://forum.doom9.org/showthread.php?s=&threadid=31301&perpage=20)

Dave

McQuaid

10th February 2003, 05:45

I also was recently dealing with that issue. I tried block buster and its seems to have solved the problem quite nicely. I did not try the luma/unfilter trick yet though. The reason I thought block buster would be better suited for this is because it is selective on where it adds noise instead of unfilter doing this everywhere.

Or am I not understanding the effect of the combination of luma/unfilter?

But even though a workaround has been found for this. It's strange the problem is there much more in mpeg4 than mpeg1. I had a tv clip that had this problem and the mpeg1 encoding barely showed the blocks whereas xvid had this problem no matter what I set the bit rate at.

MaTTeR

10th February 2003, 05:52

Originally posted by McQuaid
Or am I not understanding the effect of the combination of luma/unfilter? Well actually none of us have yet to figure out why it works so well. lol

But even though a workaround has been found for this. It's strange the problem is there much more in mpeg4 than mpeg1. I've been scratching my head about the same thing. All the MPEG-1 & 2 encoding I've done never really show the blocks during playback but it's a constant battle with all the MPEG-4 codecs :confused:

lamer_de

10th February 2003, 19:27

Here's the thread everyone's referencing.
It's well worth reading and is a who's who of doom9 personalities.

But it is aswell 14 pages of tech talk. I don't have either the time nor the concentration to read up such a huge text just to find the solution on page 10 of 14. I have no problem with reading 10 pages of "me too" or "Great work", but such threads where every post could contain the information and which deal with very technical settings are just over my head. This is aswell true for some other popular threads, which reach up to 30 pages or more.
Sure, the information is there, but a more "user-friendly approach" (you could call it aswell "cheap solution for the lazy bums") wouldn't hurt in my eyes (like in this thread, it saved me reading the 14 pages). Maybe if a solution is found on such topics, it could be added to the first post or made a link to the post which contains the solution. Or made a sticky for some time?

Only my 2 cents,
lamer_de

bobby8798

10th February 2003, 20:59

It would be nice if threads like that were deleted, and the conclusions / solutions drawn be added to an "XviD Tips & Tricks"
Sticky Post

Koepi

10th February 2003, 22:36

Delete all that knowledge? Are you serious?

Sometime later we might need such infos.

Not believing what I read,
Koepi

Belgabor

11th February 2003, 00:09

But this part

Originally posted by bobby8798
solutions drawn be added to an "XviD Tips & Tricks"
Sticky Post

is actually a good idea (imho).

McQuaid

11th February 2003, 00:10

Well, I've been trying SansGrip's Blockbuster a little more and have had amazing results with this setting:

Blockbuster(method="dither",block_size=3,luma_offset=-2,luma_threshold=30,detail_min=1, detail_max=30)

Although, I saw someone else post the block_size=3 and gave it a try, but I think it should divide evenly into the frame, no?

On the encode I just tested of a TV cap of a cartoon, I had to look out for dark blocks watching through my TV, while normally they are so glaring it's unacceptable.

Dither seems to be the key, as it stop the blocks from having a 'life of their own' (i.e. moving around or flickering where the image is relatively static).

SansGrip mentions in the readme that dither is experimental so maybe this can still be improved.

The place where I see blocks the most now is in somewhat bright solid colour areas. Instead of increasing the variables, I wonder if blockbuster can be modified to tackle a frame at both ends of the luma scale?

Anyways try it out and report back your findings.

Get his plugin here. (http://www.jungleweb.net/~sansgrip/avisynth/)

Teegedeck

11th February 2003, 07:34

Give the 'sharpen'-method (instead of 'dither') a try; very nice, too.

drebel

11th February 2003, 13:10

Removing existing mpeg2 macroblocs and not reproducing mpeg4 marcoblocks are two completely different things...;)
Blockbuster performs the first one, lumafilter+unfilter make sure about the second
I'd go with Teegedeck on this one : "sharpen" seems to be the key for me
regards,
george

PS: u could try the luma_offset=-2 option of blockbuster with unfilter(+x,+x) as a combination of both :D

McQuaid

11th February 2003, 21:51

Ok, I'm trying out sharpen right now to see the results.

Regarding luma filter/unfilter.

Almost every post I've seen is from someone using the luma function that's part of mpeg2dec3.

I'm doing tv captures so obivously, I am not using mpeg2dec3. How do I offset luma without it, if I'm not using blockbuster.

I saw one post using lumafilter(), but I can't find anything on this. Is this part of aviysynth 2.5? I'm still using 2.07.

Koepi

11th February 2003, 21:55

*sigh* even if you do tv captures:

LoadPlugin("mpeg2dec3.dll")
LumaFilter()

Read more about avisynth plugins in the avisynth forum.

Koepi

McQuaid

11th February 2003, 22:21

Doh! I was going to try that.

thx Koepi

McQuaid

11th February 2003, 22:28

Actually, I just tried it but it's giving me an error saying unknown cmd for lumafilter.

I downloaded mpeg2dec from this thread

http://forum.doom9.org/showthread.php?s=&threadid=44308

as I couldn't find a version for 2.07 anywhere else.

In the readme for mpeg2dec there is mention of a param lumoff but no mention of lumafilter.

I notice in most people scripts they use the param when loading the mpg2 file. So I'm not sure what I'm doing wrong.

Koepi

11th February 2003, 22:43

LumaFilter is only in avisynth-2.5-compiles of mpeg2dec3.dll if I'm not mistaken :-/

Maybe you should switch over ;)

Koepi

trbarry

12th February 2003, 02:30

I don't really get this problem anyway but I was playing with custom quant matrices today and noticed something I thought was interesting. The default quant number for the DC component(top left entry) in the Inter quant matrix is 16.

That seems high because whatever the DC component does spreads over the entire block, luma or chroma. Has anyone ever verified the effect on the black blocks problem using custom quants? That is, use a custom quant matrix and set the top left Inter matrix value back to 8?

Just a maybe dumb idea.

- Tom

Didée

12th February 2003, 09:21

I fiddled around with custom matrices lately. But I was looking for completely other things, not black-blocking ...

However, I found the following (bug?):

After several tests with custom quants, I must re-install XviD.
Why? Because, it (sometimes) happens that switching back to "normal" mpeg-quantization does not have the desired effect: somehow the last custom quantization table gets "locked".

If that happens, I confirm it the following way:
- Hit "load defaults"
- encode something with mpeg quants
- re-install same XviD build, hit "default"
- encode again with mpeg quants

The results of both encodes are not the same then, they have different size ...

Is someone able to reproduce that, er, feature?

Didée

12th February 2003, 13:17

Originally posted by trbarry
I don't really get this problem anyway but I was playing with custom quant matrices today and noticed something I thought was interesting. The default quant number for the DC component(top left entry) in the Inter quant matrix is 16.

That seems high because whatever the DC component does spreads over the entire block, luma or chroma. Has anyone ever verified the effect on the black blocks problem using custom quants? That is, use a custom quant matrix and set the top left Inter matrix value back to 8?

Just a maybe dumb idea.

- Tom
Yes and No - it's not that easy, alas.

The problem is that the frequency of a table's cell is not responsible/corresponding to specific content of the frame, but to the (motion-corrected) difference to ... etc. etc., you know.

So, if you simply set the very upper-left to "8", you are influencing dark/black/flat areas, indeed.
BUT you are also influencing blocks with more or even high detail, that simply didn't change that much from the preceeding frame.

Some short tests gave me an increase of filesize by 10%-20% by only changing that upper-left "16" to "8" ...

I believe that there still lies some unrevealed magic in that quantization stuff, but it's hard to find the right way and balance!

Didée

Ewi

12th February 2003, 20:32

Could it be a good idea to make a thread about this sticky? Since not only 2 or 3 have problems (me, too) with this strange thing it could be worth making it sticky. Perhaps a short answer how to prevent this (LumaFilter() / UnFilter()), what the reason is and if or when there will be a fix in Xvid itself (status of this thing at all). Otherwise there could be a lot more of threads like this, perhaps...

angelyote

12th February 2003, 22:29

Kilg0r3 has already done this in the XviD Q&A sticky. So don't read any further on my post please. No, really, there's nothing to see here. These are not the droids you're looking for.

I'd also like to see a sticky made for 'Hints & Tips' or something similar. Perhaps in a format that gives the hint and then the link to the knowledge base under it.

ie:
To help keep pure black from looking blocky try:
loadplugin(mpeg2dec3.dll)
loadplugin(unfilter.dll)
mpeg2source(yourmovie.d2v)
lumafilter()
unfilter(-,-)
http://forum.doom9.org/showthread.php?s=&threadid=31301&perpage=20

To help prevent injury:
don't run with scissors
http://www.uoguelph.ca/mediarel/archives/001249.html

Dave

yaz

13th February 2003, 12:10

Originally posted by Ewi
Could it be a good idea to make a thread about this sticky? Since not only 2 or 3 have problems (me, too) with this strange thing it could be worth making it sticky. Perhaps a short answer how to prevent this (LumaFilter() / UnFilter()), what the reason is and if or when there will be a fix in Xvid itself (status of this thing at all). Otherwise there could be a lot more of threads like this, perhaps...

motto : blocks are always there but only u're too far from the screen.

hi ewi (& all suffering from this :-)

time to time somebody opens a new thread about this topic which indicates (at least for me) that there's no unified solution of this problem. some ways work fine for some set of movies but not for others. i experienced a lot with it but i can't give a 'universal' solution (& imho, nobody else can)

anyway (kick me if i'm too lame here :-):
- blocking/tiling is inherent to all mpeg4 codec. so, u can't avoid/prevent but just minimize it & the 'minimum' is (of course) quite subjective. but (of course) there is a level unacceptable for anyone.
- this phenomenon has (can have) many different sources from hardware problems up to an improper player/postproc setting. of course, faulty filtering/encoding is also included. chasing down is quite tedious but worth to do. the most important sources are listed in the thread indicated above. check it out!

keep some things always in mind :
- improper hardware/software settings may degrade even the most perfect encode. make that clear 1st!
- removing details increases compressibility but increases (may increase) the chance of blocking. it means, e.g., never apply more filter than necessary & tune them very carefully! don't use heavy guns just for sure! sometimes, noise may be your friend not your enemy!
- u must learn (as much as possible) how the tools work. after 60(!) ugly encodes i noticed a misused parameter at a filter. if i read the manual a bit more carefully. yeah, a banging the wall.
- video encoding is a kinda lossy compression. some extent of degradation is unavoidable. we must live with it. say, the price of having a 2h movie on 1cd is some blocking here&there. does it worth or not is up to u.
- sometimes give yourself a break. lean back, take a drink & watch your encoded movie as 'normal watchers' do. u'd be surprised :-)))

i hope it helps u a bit
yaz

Ewi

13th February 2003, 18:43

Thank you for your post Yaz, but it was perhaps really a bit too lame...

That block based coding is inherent to MPEG4 is clear, I think; just like all the other DCT based coding schemes (In our compartment(university) was money freed for a new project of finding wavelet bases that are better suited than the actual ones for video compression (you have to know that there is not THE wavelet basis and all of them have different properties; in the last years there was a kind of competition finding new ones and we try to find bases that have properties that are esspecially suited for video comp); this project (and some others) are highly "observed" by the MPEG-C, so perhaps we can get rid of this block coding at all; but thats future...)

But I think esspecially Xvid has a problem with it. I didn't know that this is in the faq. My fault, so excuse me for being lazy... But what I miss there is a conclusion what's the reason and if somebody is on this issue to solve it (or not, whatever). Simply some kind of status info if something is happening to this or not, just informal...

Thank you for your answer...

bobby8798

13th February 2003, 19:06

Originally posted by Ewi
That block based coding is inherent to MPEG4 is clear,...

But I think esspecially Xvid has a problem with it

XviD is seriously deficient in that area, but the problem should be solved over time. IF the problem is extremely bad for a particular movie, just use DivX for the meantime.

Just a NOTE:
Be careful when using filters to get rid of the blocks. I just noticed someone with dark brown, or black hair will get all the details REMOVED. Sofas, Tables, and dark furniture are susceptible also.

The LumaFilter works well for the opening and end credits.

Ewi

13th February 2003, 20:07

Originally posted by bobby8798
Be careful when using filters to get rid of the blocks. I just noticed someone with dark brown, or black hair will get all the details REMOVED. Sofas, Tables, and dark furniture are susceptible also.

Exactly. Thats always (when it happens) my problem. It looks not only blured; it looks like if it was masked and "floodfilled" (as you know it from photoshop); the result looks very cold and computer-drawn (in these areas). I don't have screenshots at the moment cause I'm testing dev-builds of Xvid and I don't want the encodes to get in the wild (I fear later decoder incompabilities), so I delete them right after testing; but I think what the problem is, is clear....

Holomatrix

24th June 2003, 15:55

I'm trying to find away to fix this moving Blocky/Black areas to and was wondering what you would recommend for doing DVD to SVCD conversions?

lumafilter()
unfilter(-,-)

or Blockbuster (dither) I aleady use Blockbuster (Noise) should I use both ?

Thanks

Koepi

24th June 2003, 16:00

I use the lumafilter() + (optionally) Unfilter(-,-) solution (still) successfully.

Regards
Koepi

Holomatrix

24th June 2003, 16:18

Ok, I'll try it out. I'm testing it with low bitrates. I did a NTSC movie called Equilibrium 106 min on 1CD at 880 bitrate and for some reason it looks very good with just Lanczos Resize no other filters(5gig to 800mg) except for in the dark sceens I get this dancing black block effect. I'll try luma and unfilter and see what happens.
Thanks

Holomatrix

24th June 2003, 16:30

Also what setting should I use for Unfilter? I tried (-,-) but the script crashed. I'm trying with (-1,-1) now, that works. Is that ok?

Koepi

24th June 2003, 16:31

Unfilter(-5,-5) is a good start.

yaz

25th June 2003, 09:05

Originally posted by Holomatrix
Also what setting should I use for Unfilter? I tried (-,-) but the script crashed. I'm trying with (-1,-1) now, that works. Is that ok?
oops ... what does 'crash' mean here? have u got syntax error, exceptional, access violation or what? just ask cus i got problems with unfilter in yuy2 permanently but in yv12 it's ok.

when using blur/sharpen tricks take care of proper yuv scaling too. the same is true for blockbuster. i've just noticed that a proper yuv scaling at the very end of the script solves most of the blocking problems. u can check the scale with coloryuv(analyze=true). it is especially important if u (want to) watch the clips on tv.
btw, i don't recommend lumafilter. use tweak or coloryuv insted.

the bests
y

kilg0r3

25th June 2003, 15:44

1.)
hi, yaz
now you can help me out :).Originally posted by yaz
proper yuv scaling

what's that mean? is this some multiple of at least 4 thingy.

2.)
@Koepi
You are actually softening the image? I always that the opposite was necessary to 'keep the quantizer active'.

Holomatrix

25th June 2003, 16:07

Originally posted by yaz
oops ... what does 'crash' mean here? have u got syntax error, exceptional, access violation or what? just ask cus i got problems with unfilter in yuy2 permanently but in yv12 it's ok.

when using blur/sharpen tricks take care of proper yuv scaling too. the same is true for blockbuster. i've just noticed that a proper yuv scaling at the very end of the script solves most of the blocking problems. u can check the scale with coloryuv(analyze=true). it is especially important if u (want to) watch the clips on tv.
btw, i don't recommend lumafilter. use tweak or coloryuv insted.

the bests
y

Ok, that's weird, yesterday when I was doing the autofitcd preview thing in DVD2SVCD I clicked test and I got some red writing in the display that was telling me that there was something wrong with my script but when I tried it today with (-,-) it was fine ??? I must have forgot somthing yesterday, maybe my loadplugin line was bad.
Anyway, is (-,-) ok to use now or stick with (-5,-5)?
Thanks

EDIT: See my dilema is that I've been using one standard for a long time now. Bicubic, fluxsmooth, Blockbuster(noise) and asharp and have been pretty happy with the results and I'm trying a few different things now so I don't fall behind in quality. I tried Lanczos resize alone on this movie and liked that Lanczos was giving me the sharpness I needed but came back to the forum looking for a new way of blending these dark blocks when I ran into this thread, and since I've been reading abit I've ran into a few to many options, which include - Deen/Undot, Lumafilter/Unfilter, tweak/colouryuv or my original way so you see I guess I need to start testing again and maybe you guys could post some updated scripts for me to try and remember I'm doing SVCD not Xvid. I'm using Avisynth 252 so I guess that would meen that my filters are working in the yv12 color space then when DVD2SVCD adds convertoyuy2 then fed yuy2 to CCE. Does that work for all the above filters? Thanks for any info

yaz

1st July 2003, 10:58

@kilg0r3
-yuv scaling (maybe not the best term)
the final conclusion drawn in that monstruos thread about black-blocking was using a combination of lumafilter, unfilter, blockbuster, ... that's ok. it's a way to get rid of that annoying blocks. but ... the (side)effect of these filters is the distortion of the y(uv) scale.
say, even if the input luma range is ok (16-235), after the unfilter/blockbuster combo it's surely not. this (alone) may cause blocking on some tv screens. all filters we have are 'clamped' to 0-255 luma but not to any tv standard. that's why i always check 'scaling' with coloryuv() while tweaking a filter chain. i make the proper yuv correction in the end. instead of clamping i prefer scaling (coloryuv, tweak, leveles, ...)
note, (most of) the filters used against blocking work only on the luma plane.

-yv12
i (must:-) agree with iago. black blocking is much less frequent in yv12. dunno why, but there it is. in add, i've never met improper color (u/v) scaling which occured frequently with yuy2.

the bests
y

Koepi

1st July 2003, 11:15

Originally posted by kilg0r3
@Koepi
You are actually softening the image? I always that the opposite was necessary to 'keep the quantizer active'.

Softening the image with unfilter has the same block-reducing effect as sharpening it. Might be the way the filter works.

After reading all this, I'll statr an encoding without all these filters in an YV12 process chain. If it's really the YUV2->YV12 conversion which is causing the issue we should look deeper into the conversion routines IMO.

Regards
Koepi