How to prevent grain induced block noise in uniform areas?

[adoniscik]

I am trying to get a 1MBps video with a smooth sky and a reasonable encoding rate (over 10 fps), but I realize this may not be possible. If not, I want to know how close I can get.

I read the archives voraciously, yet I could not solve the problem. Some of the suggestions contradict each other! Some people say block noise is due to bit starvation, so noise should be added. Others say it is a result of noise, and should be filtered out. What is a poor soul to do?

Since there was so much grain to begin with I felt no need to use the dithering approach. That leaves the noise reduction approach, of which I tried the following Avisynth 2.5 filters to no avail, used after de-interlacing with tomsmocomp(1,5,1):

mpeg2source with the cpu=6 parameter.
Convolution3d(preset="movieLQ") (as well as other parameters)
peachsmoother(NoiseReduction=100)
smoothhiq(7,20,20,256,0)
VagueDenoiser(threshold=0.8,method=1,nsteps=6,chroma=true)

The only thing that remotely helped was enabling "Film Effect" in the XVID decoder.

Xvid 1.0 parameters:

AS @ L5, Two-pass @ 1000kbps
"Ultra High" motion precision search
"Mode decision" VHQ mode
Default quantizer restrictions: 1,31,1,31,1,31
Trellision quantization
MPEG quantizer (but I tried 'em all)

[Asmodian]

Try with no trellision quantization. Also a quant range of 2-31 and the h263 or HVS Best matrix might help.

EDIT: Do you just use standard B frame settings?

[*.mp4 guy]

try these settings:

custom mpg matrix sooulhuntersv5:  " "$ "$0 "$08 "$08@ "$08@€  $ $* $*0 $*08 $*08@ $*08@` $*08@`€

gmc
interlaced
qpel

default bvops
packed bitstream
closed gov

chroma optimizer enabled

motion search6
vhq4
use chroma motion
I frame interval 300

min quant 2 for all frames
trellis quant

to use the custom matrix just copy paste it into a txt file and open the txt file under the load custom matrix section.

[iago]

Blockiness on smooth surfaces, which contain little detail... A widely-discussed and never ending problem. Filtering out the grain only makes things worse in such situations.

Do not filter out the grain and use a matrix that helps keep details. (Since you say the film-effect -adding grain- during playback doesn't disturb you.) If high quantizers are used in that scene, use "zones" feature to assign lower quants to that particular section.

[Manao]

adoniscik : blocks are indeed due to overquantization of relatively smooth areas. This overquantization happens when you don't have enough bits to properly encode a frame. It is mainly visible in smooth area, because blocks are more visible in those areas.

So, a way to fight it is to make those areas less smooth, that's why some people were advising to add noise. But there is a counterpart : if you add noise, you're making the movie less compressible, so you'll lower the overall quality of the movie. The trick is to add enough noise to make the blocks almost disappear, and yet retain enough quality for the rest of the movie.

However, I would tend to use postprocessing to get rid of blocks. I don't like the idea of making the movie less compressible ( but that's a personnal opinion ).

Now, for XviD's settings, I don't know whether you used b-frames or not, but you should use them. As Asmodian say, you should try without Treillis quantization, because one of its effect is to discard useless DCT coefficients, which alas creates such blocks. You should definitely try a custom matrix ( I like HVS-Best )

[Teegedeck]

I do agree with iago and Manao. Though I'm not sure about disabling Trellis. Because Trellis has very good effects that I wouldn't want to miss. For example it gives a lower filesize while contours seem to remain more accurate with it than witout it, and in most cases it seems to prevent more quantization errors than it creates...

As iago said, first try to quantize the problematic zone less, use hvs-best as Manao said. If that doesn't help, add noise with that avisynth filter, err, I believe its name was "blockbuster", but only for that zone.

[Lord_KiRon]

adoniscik - are you by any chance try to play your clip on MTK based hardware MPEG4 player ? - This players are known to produce a lot of blocks when displaying uniform areas , when on PC they are noticable only if you set high resolution .

[adoniscik]

No, I am using the standard PC players; BSPlayer, MPC, and Media Player. I did use B-frames, at the Xvid 1.0 standard settings.

I ended up using noise reduction and enabling "film effect" in the decoder, but keep the suggestions coming. I am sure someone will benefit from them. Out of curiosity, does this problem exist with wavelet compression?

[Prettz]

The best solution is to completely and totally eliminate all noise in the still areas. If you can do this, you will (obviously) eliminate the problem. However, usually this is not possible, and in some cases attempting to remove the noise just makes the compressed result look even worse than before (especially when you try to remove the noise with Convolution3D).

In cases where you can't eliminate the noise, you should just do some light filtering (with filters like FluxSmooth and Undot) and use zones to up the bitrate, lower the quantizers, and reduce the number of bframes used. At least, this is the solution that has worked best for me in the past.

[Manao]

The best solution is to completely and totally eliminate all noise in the still areas. If you can do this, you will (obviously) eliminate the problem.No. Doing so, you'll end up with a smooth gradient, which will be overquantized by XviD, because there isn't enough detail in it. Overquantization will break the smooth gradient ( in a sort of step-function ( if that word exists ) ), hence creating blocks. Such denoising will only prevent them from moving.

[eb]

adoniscik,

I processed your sample, you can find it on

ftp://www.eb.enterpol.pl
user: www.eb.enterpol.pl
password : eb

name of the sample ss1++.avi

eb

[adoniscik]

I downloaded your version. Thank you for the effort. The blocking is gone, but I notice ghosting, presumably due to aggressive temporal smoothing. Do you see it too? Judging by the 4CC, I think you used 3ivx rather than xvid.

[eb]

Yes 3ivx was used.
But most weired in your sample is shaking picture so I used deshaker plugin, then smoother,denoiser and again smoother, all this in the second pass of deshaker.
eb

[adoniscik]

Is that Gunnar Thalin's deshaker (http://biphome.spray.se/gunnart/video/deshaker.htm)? I would imagine stabilizing affects compressibility because shaking makes motion estimation more difficult. I will investigate this. What is the performance penalty?

[lordadmira]

Yeah, it's a common problem u have there. But if you ask me trying to get 1 Mbps is pushing it for that frame size. Nevertheless I gave a go at it, I reused most of the settings I use for encoding my Inuyasha eps. :) I got a 504kB clip that looks much better than eb's attempt (overall). He used Quant 1 which is insane. ;) I uploaded the config file too so you can check out the settings.

The problem with fine gradients like that is like Manao said, it's a problem with the dynamic range. However I don't see that problem in ur clip. The sky is chaotic enough to encode fine. It's the *really* fine gradients that have the potential to break the codec. Any blockiness u see is do to too low a bitrate. My encode went to about 820 kbps, 1000 is too high for this particular scene. I did a Q=2 encode and it only came out at under 1300 kbps. There is some choppiness to the sky (choppiness is diff from blockiness), that's normal for this codec for the dynamic range considerations already discussed. This is because the codec works by forcing square pegs into round holes, there's limitations to what that can achieve. Even Q=1 in my tests gave minor choppiness. To get any better u need custom matrixes. That's a trial and error process. The default matrix is limited in what it can do. I used Virtual Dub Mod and Smart Deinterlacer. The other filters are built in.

http://w3.goodnews.net/~wagnerc/adonisck.vcf

http://w3.goodnews.net/~wagnerc/adonisck[Xvid_1000].avi

LA

[lordadmira]

Originally posted by Manao
No. Doing so, you'll end up with a smooth gradient, which will be overquantized by XviD, because there isn't enough detail in it. Overquantization will break the smooth gradient ( in a sort of step-function ( if that word exists ) ), hence creating blocks. Such denoising will only prevent them from moving. Eh, sort of. What we call a fine gradient is an area that has a small dynamic range. It's a range and pattern that doesn't correspond well to any of the quant matrix's transforms. So the codec has to force the area to take on one of set of bad matches. The result is choppiness. Blockiness is when the edge patterns of blocks don't line up/match up. The codec is optimized to handle well a certain range of patterns and gradients that occur in natural scenes. Fine gradients break that at one end of the spectrum and anime breaks it at the other end. Solid color -> black line -> solid color is not something it likes to deal with. It's the same phenomenon that causes the codec to go into rate control anarchy when it starts using quants 1's and 2's. Math might tell u that 1.52987 is the right choice, but sorry! U have to pick 1 or 2. Same with the fine gradients.

@iago
I don't think the amount of noise makes any difference to the dynamic range problem. Because although the noise will expand the absolute range, it will not change the average range. This is why temporal-spatial smoothers are able to filter out noise in the first place. So adding noise will just add to the blockiness problem since bits have to be spent to handle that large absolute range and do nothing for the choppiness problem which is one of average dynamic range.

LA

[Manao]

lordadmira : I think we're speaking of the same thing. With low quant, you can approximate very well ( to a certain extent ) all the values taken by DCT coefficients, while at higher quants, a stronger rounding will happen. When you round too much the coefficient that represent the mean value of the block ( I don't remember whether it is DC or AC, let's call it DC ), you obtain the block effect : DC coefficients will differ from one block to another, and continuity won't be brought back by other coefficients, because they are even more quantized.

[lordadmira]

Heheh, yeah, choosing the right words gets hard in these esoteric discussions. ^_^ U need to be a damn English major sometimes...

[manono]

adoniscik-

You've got a bad PAL to NTSC DVD there. Part of your problem stems from the fact that you're just deinterlacing it (and filtering the hell out of it) and keeping it at 29.97fps. If you were to use the KernelBob/RePAL combo, it'll become 24.975fps, distribute the available bits among fewer frames, have higher quality with a lower average quant for the same file size, and generally give you a much better looking result.

[eb]

lordadmira wrote
I got a 504kB clip that looks much better than eb's attempt (overall). He used Quant 1 which is insane.
Yes you are right, I used this to make quick assessment what max bitrate is needed to not to loose details, then I forget about this, jumping to the deshaking problem.
And I must underline that the prime target of my post was to take care of adoniscik to the shaking problem.

lordadmira:
Heheh, yeah, choosing the right words gets hard in these esoteric discussions. ^_^ U need to be a damn English major sometimes...
so what I can say with my crappy English

eb

[adoniscik]

You've got a bad PAL to NTSC DVD there.
Why did I not think of that? I saw the blended frames, but I did not infer the presence of telecine. You learn something new every day! If I have any questions about the tutorial (http://www.doom9.org/ivtc-tut.htm) I know where to ask.

[lordadmira]

You have to be very careful dealing with inverse telecining. U can only successfully do it if the movie was telecined properly in the first place. I've run into videos that were constructed from various telecined pieces stitched together. The IVT starts off in sync but then hits a stitch point and the sync is lost. Stuff after that point will have strange interlacing artifacts due to the out of sync IVT. IVT only works right if the whole video was telecined after being made. The only real way to tell this is to load it up in VDub with the IVT and frame through it looking for artifacts. It's a real pain.

Looking at ur video it seems not to be telecined but truely interlaced. Doing an IVT left visible interlacing artifacts.

edit: Looking again it appears 1 in 6 frames is non-interlaced. This could indicate some strange film -> PAL -> NTSC conversion and some more bizarre telecining process. Undoing that much butchery might be more trouble than it's worth. Straight deinterlacing gave me good results.

eb: LOL :)

[Lord_KiRon]

Can't the process be "automated" ? I mean it shouldn't be to hard having the process(algorithm) to detect telecine to enhance it to detect videos that were constructed from various telecined pieces , after that we have only need to make decission on that to do at stich point .

[drcl]

use donald graft's decomb for avisnth.

it has a reset feature especially for this problem - you'll need to read the help file


I reckon your initial problem is noise induced. remove it with a high radius using temporal soften


temporalsoften(8, 2 or 3, 8, 16, 2)

[drcl]

Close examination of the clip show a fair amount of chroma noise. you might want to try cnr2()

[Prettz]

Originally posted by Manao
No. Doing so, you'll end up with a smooth gradient, which will be overquantized by XviD, because there isn't enough detail in it. Overquantization will break the smooth gradient ( in a sort of step-function ( if that word exists ) ), hence creating blocks. Such denoising will only prevent them from moving.
I completely disagree with this. When I said "completely and totally eliminate all noise in the still areas" I meant the best case scenario when you can do this without smoothing or removing detail. This happens occasionally.

If you can't remove random noise, you can often end up with just larger solid areas of random noise moving about, which look absolutely hideous in the final encode (this happens a lot with animation). In these cases you're best off trying to reduce the sharpness of the noise as much as possible without just smoothing the noise around.

[Manao]

Prettz : as soon as I get home, I'll create a noiseless gradient, and test my assertion ( which is that if I overquantize that noiseless gradient, I'll get blocks ). But I'm almost sure to be right.

[lordadmira]

Of course it's possible to crush any image type (including a pure gradient) enough to get blocks. The question is how much will Xvid decide on it's own to quantize fine gradients in the context of the video overall. I think ur point is that fine gradients will break the codec more readily than general images. In these cases u need a custom matrix to bring more quant numbers into the range that the gradient "wants" to quantize to.

LA

[Manao]

I did the test : I created a gradient going horizontally from deep black to deep white ( luma increasing by one with each pixels ). I encoded it, with XviD, const quant 4 and 7, H263 ( I also tried quant 3, and HVSBest quant 3, results were almost the same as H263 quant 4 ), without fancy options. The pictures shown are keyframes, but the followings are almost the same ( since there is no motion )

http://jourdan.madism.org/~manao/gradient-original.png

http://jourdan.madism.org/~manao/gradient-h263-q4.png

http://jourdan.madism.org/~manao/gradient-h263-q7.png

You should see that the gradient isn't smooth anymore, even at quant 4 ( which can't be called image crushing, imho )

[Teegedeck]

:goodpost:

[lordadmira]

That's a really good demonstration of the limitations of DCT encoding. And actually, that is an even steeper gradient than the ones that give us problems in real life encodes. I'ld be interested to see what quants 1 and 2 look like. Hmm, what gradient would be considered "the worst case scenario" for Xvid? We should save that and add it to a FAQ somewhere.

PS Please forgive my cavalier use of the word "crush". :D

[Manao]

Hmm, what gradient would be considered "the worst case scenario" for Xvid?It will be most visible on artificial gradients, because the eye expect something XviD can't reproduce. On natural gradient, it's less visible.

Another thing : I made screenshots with VDubMod, and something strikes me : even with the original gradient, we can see a sort of blocking ( very faint, only visible if zoomed enough ). I wonder whether a convertion in RGB wouldn't be the culprit of this one.

And for the third screenshot, am I the only one to see faint green lines ? The AVS script was a blankclip() followed by a filter which touched only the luma channel, so chroma should be uniform, so XviD shouldn't show such green lines. Is that an encoder or decoder bug ( if it is even a bug ) ?

Finally, I also tested quant 2. Keyframe is slightly blocky ( though there it may be RGB convertion the culprit ), and, more surprising, not vertically uniform. However, the following p-frames tend to correct both blocks and uniformity ( though it needs 5 p-frames ). The same effect ( disparition of the blocks ) was also almost achieved after 7-8 p-frames at Q4, and but not at Q7.

[lordadmira]

But which artificial gradient would be the worst case scenario? You probably had 16 bit color on the preview windows. That's the default unless u change it. There's no problem going between YUV and RGB. (Not counting YV12 which is evil!) I didn't see any green lines on #3 but there is some strange color-like mottling. That could be anything from a compression artifact to a moire illusion. I think there is some Xvid bug where it can't handle pure B&W material.

[Soulhunter]

Hmm...

Loading the pic as uncompressed Avi in VDub via AviSynth...

AviSource("C:\Blah.avi")

http://img6.imageshack.us/img6/5669/00001.png


Loading the pic as uncompressed Avi in VDub via AviSynth...

AviSource("C:\Blah.avi").ConverttoYV12()

http://img17.imageshack.us/img17/6972/00004.png


Bye

[lordadmira]

Originally posted by Soulhunter
AviSource("C:\Blah.avi").ConverttoYV12()
Ah, sheer beauty. A clear demonstration of the evil of the YV12 colorspace. :devil:

LA

addendum: I opened up the "original" gradients in Paint Shop and there are only 219 colors present. The YV12 gradient only has 189 colors.

[Manao]

Soulhunter : this should not be relevant, since the output of my avs script is YV12, and XviD also encodes to YV12.

What bothers me, however, is that if I open the avi directly with virtual dub, or with avisynth, ( AVISource("foo.avi", pixel_type="YV12") ), I get different results.

lordadmira : I don't think ( alas ) that it's a moire illusion. It is not a problem of VDub being in 16 bits ( it was in 'desktop color depth' mode ). And for black & white, YV12 is not evil.

Edit : for the number of color : somebody ( something ) put a limiter somewhere ( hence reducing the luma range to 16-235 which, omg, makes only 219 available grey levels )

[RadicalEd]

The green lines seem to come from the non-mod-16 height. Crop it down to 256x96 and I'd bet it'll work.

[Manao]

RadicalEd : alas, no. I encoded a 720x544 clip, and I cropped after having made the screenshots. I should have mentioned this, sorry.

[RadicalEd]

Strange. I encoded one sample at 256x96, another at 256x90, and the same kind of lines were present (only) in the latter.

[lordadmira]

But there are fewer grays in the YV12 sample than in the normal sample. How do u explain that. I know it "shouldn't" matter.

@ed Going non mod16 shouldn't make any difference.

[Manao]

lordadmira : RGB -> YV12 conversions are lossy. Rounding are made. So it may happen that the number of color is reduced.
Look at the following example :

Suppose that for conveting from a monocolorspace X to another monocolorspace Y, you have to multiply the value by 2. Now, you have a gradient going from 0 to 255 in the Y colorspace. You convert it to the X colorspace ( so you're dividing by 2, and rounding ), then back to the Y colorspace. You now have the following gradient :

0 0 2 2 4 4 6 6 .. 254 254

That's a crude example, but that's roughly what is happening when converting from RGB to YV12.

[lordadmira]

But for a grayscale image a YV12 conversion shouldn't be lossy. It is lossy for color images. This is because only the UV planes are subsampled. The Y plane, the only one with nonzero data in a grayscale image, is full rez. So if the chroma info is blank, as it should be in these pics, a YV12 conversion shouldn't degrade the image.

@ed I don't know what green lines ur talking about. I ran the color dropper over the third image and nothing that could be considered green came up. Almost everything was a pure gray, a couple lines had a slight excess of green but only by 3 or 4.

[RadicalEd]

You're not looking hard enough. The discontinuities are definitely visible, zoomed or not. Even so, no mathematical discontinuities should be present between RGB channels.
[edit]FYI, the first one goes from y = 6 to 9

[Prettz]

Eh, I never mentioned anything about creating a gradient, only removing noise. In my experience the gradients are often destroyed not by the encoder but by the noise removing filters themselves (again, especially Convolution3D).

I think Xvid generally handles gradients very well, and orders of magnitude better than Divx 5.

[Manao]

But for a grayscale image a YV12 conversion shouldn't be lossy.I just looked at the source code, and, at least for the C version, it's lossy. Basically, the luma channel is computed by the following formula :

((0.114 * r + 0.587 * g + 0.299 * b) * 219 / 255 + 16

Hence, if we had 219 different grey levels, after making a conversion to RGB, then back to YV12, there'll be 219 * 219 / 255 = 188.08 different grey levels.

a couple lines had a slight excess of green but only by 1 or 2.So I wasn't dreaming. I don't see what could mathematically create these lines. It means that DCT coefficients ( other than DC ) are different from 0, where there should have been zero. But since input in chroma is homogenous, I can't see how it would happen ( except, again, a bad rouding, but since it happens only at higher quants, there is something wrong ).

[lordadmira]

Oh ur talking about *horizontal* lines. I was only looking at the vertical shading lines. :D Now I can see the slight horizontal green lines. They're made up of pixels that are 3 points higher in green than pure gray. They are 4 pixels high and go through the middle of the macroblocks. As for the vertical discontinuities, I think those are illusions since I examined it with the dropper and way zoomed in. They're just different widths of the various gray lines and the fact that the gray is the wrong shade.

[lordadmira]

Originally posted by Manao
I just looked at the source code, and, at least for the C version, it's lossy. Basically, the luma channel is computed by the following formula :

((0.114 * r + 0.587 * g + 0.299 * b) * 219 / 255 + 16 That is seriously screwed up. What program is that? AviSynth? Are u sure that's not for the YUY2 colorspace used by NTSC television? The Y channel must have a valid range of 0 - 255. All this crap is why I say to hell with non RGB colorspace. :devil:

I think this green line has to do with a semi known bug where Xvid pukes on B&W material. Some people see green, some see pink. I've had B&W encodes look pink.


Edit: NTSC uses the YIQ colorspace.

[Manao]

What program is that? AviSynth?YepAre u sure that's not for the YUY2 colorspace used by NTSC television?It the C function called RGB2YUV, which convert a rgb pixel into a yuv one. But I don't know how works the real converttoyv12(), since it's all in asm / mmx.
I think this green line has to do with a semi known bug where Xvid pukes on B&W materialI always thought it was because the iDCT had changed many times in the early releases of XviD.All this crap is why I say to hell with non RGB colorspace.If only screen were made of YUV pixels instead of RGB ones. I don't like RGB because it doesn't take into account how works the HVS.

[lordadmira]

Well YUV was invented as a hack when color TV's came out so that black and white TV's would still work with the new signals. It used to be that there was just the "Y" signal. Instead of making a new transmission scheme they cleverly just "added" the U and V signals so that the new TV's could be full color and the old ones would still work. We've been paying the price ever since....

LA

[virus]

Originally posted by lordadmira
We've been paying the price ever since....Actually YUV is not a burden to carry on for historical reasons, but a way to provide better compression. The YUV components are (almost) decorrelated whilst the RGB ones are not. That means less variance for the U and V planes, hence less entropy, thus they can be represented with less bits.
All serious pictures/video compression codecs use some YUV colorspace (not always the same) tu reduce spectral redundancy.